Software Engineering - Monitor program to capture bugs?

Author

Monitor program to capture bugs?

sheldon.nicholl@state.or.us

2006-09-12, 7:02 pm

Currently, users of some of our programs encounter mysterious bugs. In
order to fix these bugs, however, developers need to be able to
reproduce them.

The problem is that developers seldom have enough information to be
able to reproduce a bug. In many cases, a whole sequence of user events
over time is required to reproduce the bug. Currently, we have no
cost-effective way to record this user information.

Even if we were able to implement this kind of recording functionality
inside an application, the record would likely be unreliable precisely
because the app has encountered a bug and is therefore in an unreliable
state. Furthermore, with thousands of applications spread across the
organization, it would not be possible to implement recording
functionality in each app.

So our desire is to find monitoring software able to record user bugs.
The monitoring software would capture what the user did prior to the
occurrence of the bug so we could reproduce it.

Clearly, such a program could be misused; we don't want to create
spyware. Ideally, the monitor program would make itself quite obvious
to the user and would only record interactions with a single specified
application. Our security department has already specified that both
the user and their manager must be notified prior to the installation
of the software and when it is removed, as well as why we're doing it
and what we're looking for.

We are therefore looking for advice and experience as to what software
would be appropriate for our purposes, as well as what controls should
be put in place firstly to prevent abuse, but more importantly, to
avoid even the appearance of impropriety.

James Bond 007

2006-09-12, 7:02 pm

<sheldon.nicholl@state.or.us> wrote in message =
news:1158076537.163437.121420@h48g2000cwc.googlegroups.com...
> Currently, users of some of our programs encounter mysterious bugs. In
> order to fix these bugs, however, developers need to be able to
> reproduce them.
>=20
> The problem is that developers seldom have enough information to be
> able to reproduce a bug. In many cases, a whole sequence of user =
events
> over time is required to reproduce the bug. Currently, we have no
> cost-effective way to record this user information.
>=20
> Even if we were able to implement this kind of recording functionality
> inside an application, the record would likely be unreliable precisely
> because the app has encountered a bug and is therefore in an =
unreliable
> state. Furthermore, with thousands of applications spread across the
> organization, it would not be possible to implement recording
> functionality in each app.
>=20
> So our desire is to find monitoring software able to record user bugs.
> The monitoring software would capture what the user did prior to the
> occurrence of the bug so we could reproduce it.
>=20
> Clearly, such a program could be misused; we don't want to create
> spyware. Ideally, the monitor program would make itself quite obvious
> to the user and would only record interactions with a single specified
> application. Our security department has already specified that both
> the user and their manager must be notified prior to the installation
> of the software and when it is removed, as well as why we're doing it
> and what we're looking for.
>=20
> We are therefore looking for advice and experience as to what software
> would be appropriate for our purposes, as well as what controls should
> be put in place firstly to prevent abuse, but more importantly, to
> avoid even the appearance of impropriety.

Have you considered logging? Many systems and applications have logging =
capabilities built into them. The logs typically go into the same =
folder where the executable files reside (Windows) or into a common =
logging directory (UNIX/Linux). The logging is typically engineered so =
that at least several hours, if not days, worth of logging exists in a =
file before it is overridden. Logging can be designed with multiple =
levels - the higher the level, the more information is logged. Typical =
log file entries include when users log in and out, when applications =
are started and stopped, when modules and functions are invoked, what =
are the environment variables, etc. When the file becomes full, it =
typically rolls over into a .old version, and the original log file is =
then cleared and started over. Systems that have multiple applications =
can have separate log files for each application if desired. =20

When users report a bug or call for support, one of the first things =
that developers or support people should do is upload the log files back =
to their office for analysis.

sheldon.nicholl@state.or.us

2006-09-12, 7:02 pm

Thanks for the idea about logging. Actually I should have specified
that the application driving this process is a very GUI-heavy app
written in ObjectStudio Smalltalk. We're looking for the logger or
monitor to give us output like:

1. User entered text: 'Daily' on screen S1.
2. User clicked on button B on screen S1.
3. Screen S2 came up.
4. User selected 'Active' from drop-down list D.
5. User clicks close button on S1.
6. Exception occurs.

James Bond 007 wrote:
> <sheldon.nicholl@state.or.us> wrote in message news:1158076537.163437.121420@h48g2000cwc.googlegroups.com...
>
> Have you considered logging? Many systems and applications have logging capabilities built into them. The logs typically go into the same folder where the executable files reside (Windows) or into a common logging directory (UNIX/Linux). The logging
is typically engineered so that at least several hours, if not days, worth of logging exists in a file before it is overridden. Logging can be designed with multiple levels - the higher the level, the more information is logged. Typical log file entries
include when users log in and out, when applications are started and stopped, when modules and functions are invoked, what are the environment variables, etc. When the file becomes full, it typically rolls over into a .old version, and the original log
file is then cleared and started over. Systems that have multiple applications can have separate log files for each application if desired.

James Bond 007

2006-09-12, 7:02 pm

I don't know how you could log how a user moves the mouse or enters text =
in a field. But when they click on a button, that action can be logged =
along with the content of all fields. A screen certainly could log when =
it comes up and dump all its environment variables and GUI fields. =
Since the GUI responds to the user selecting a value from a drop down =
list box, that could easily be logged as well. If an exception occurs =
after a user clicks a button, the logging of that action should be the =
first thing your application does when the button is clicked. If an =
application or function is suspect, have more detailed logging such as =
dumping the contents of all local variables and when every internal =
function is called.=20

<sheldon.nicholl@state.or.us> wrote in message =
news:1158097761.187151.251410@b28g2000cwb.googlegroups.com...
> Thanks for the idea about logging. Actually I should have specified
> that the application driving this process is a very GUI-heavy app
> written in ObjectStudio Smalltalk. We're looking for the logger or
> monitor to give us output like:
>=20
> 1. User entered text: 'Daily' on screen S1.
> 2. User clicked on button B on screen S1.
> 3. Screen S2 came up.
> 4. User selected 'Active' from drop-down list D.
> 5. User clicks close button on S1.
> 6. Exception occurs.
>=20
>=20
> James Bond 007 wrote:
news:1158076537.163437.121420@h48g2000cwc.googlegroups.com...[color=darkred]
logging capabilities built into them. The logs typically go into the =
same folder where the executable files reside (Windows) or into a common =
logging directory (UNIX/Linux). The logging is typically engineered so =
that at least several hours, if not days, worth of logging exists in a =
file before it is overridden. Logging can be designed with multiple =
levels - the higher the level, the more information is logged. Typical =
log file entries include when users log in and out, when applications =
are started and stopped, when modules and functions are invoked, what =
are the environment variables, etc. When the file becomes full, it =
typically rolls over into a .old version, and the original log file is =
then cleared and started over. Systems that have multiple applications =
can have separate log files for each application if desired.[color=darkred]
>

JXStern

2006-09-12, 9:59 pm

On 12 Sep 2006 08:55:37 -0700, sheldon.nicholl@state.or.us wrote:
>Currently, users of some of our programs encounter mysterious bugs. In
>order to fix these bugs, however, developers need to be able to
>reproduce them.
....

Do you have error handlers on ALL events handlers?

Just have them all call a common state dump routine to a local seq
file or database table.

Over the years I've seen packages come and go that could hook this
stuff in, rather than make you code it all, but I don't know of any
currently.

Probably never seen a generic package that could catch, report, AND
recover, though, that takes coding. It's a best practice.

J.

H. S. Lahman

2006-09-14, 7:00 pm

Responding to Sheldon.nicholl...

> Even if we were able to implement this kind of recording functionality
> inside an application, the record would likely be unreliable precisely
> because the app has encountered a bug and is therefore in an unreliable
> state. Furthermore, with thousands of applications spread across the
> organization, it would not be possible to implement recording
> functionality in each app.

Logging of some sort is the only way to provide a program trace of the
flow of control within the program. Basically there are three ways to
do that: provide a wrapper environment that acts like a debugger and
steps through the program; add a module that records the application's
call stack (provided the application has sufficient privilege to access
it); and embedded print statements that log identity, inputs, and
outputs for every procedure to a file.

Some shops provide embedded print statements to a log stream as standard
part of the initial development. Usually they provide a separate
debugging build with the statements while not compiling them for a
production build. IMO that is an excellent practice for large, complex
applications because it doesn't cost much to do with macros and whatnot
when originally coding the procedures but it pays huge benefits in time
saved during development and after release for exactly these sorts of
problems.

The main di

vantage of loggers is that they tend to product gigantic
log files. So it is usually a good idea to provide multiple levels of
logging. If such a logging facility is carefully designed initially,
one can even turn logging on and off at particular points in the
execution. Alas, I gather you do not have such a standard. B-(

>
> So our desire is to find monitoring software able to record user bugs.
> The monitoring software would capture what the user did prior to the
> occurrence of the bug so we could reproduce it.

I would google for this. There are a number of GUI monitoring tools on
the market that can be inserted between the OS Window Manger and the
application to provide logging of the Window Manager messages.

The problem (unless the technology has improved since I last used one)
is that such loggers are very low level and one has to reconstruct
things like form fields from individual characters. It might be a lot
easier to provide a logging facility inside your GUI access subsystem to
log user actions at the Window/Control level after the WM_XXX messages
have been decoded.

Note that there is a third choice that you did not mention. Today one
can execute a program remotely where the user's GUI appears on the
customer support person's terminal. That's great for catching mistakes
in the way the software is used that the user didn't describe. But it
is probably a waste of the support person's time to wait around for
intermittent software bugs. However, one could use a variation on this
to log keystrokes remotely so that they don't fill up the user's disk.
That is, the user application logs into a remote monitoring facility.

> Clearly, such a program could be misused; we don't want to create
> spyware. Ideally, the monitor program would make itself quite obvious
> to the user and would only record interactions with a single specified
> application. Our security department has already specified that both
> the user and their manager must be notified prior to the installation
> of the software and when it is removed, as well as why we're doing it
> and what we're looking for.

I think such security is more of a problem of user perception than
reality. You can place logging systems' ON/OFF under user control and
provide an UNINSTALL for any external software that you temporarily
provide. For non-remote logging the user can verify that it is off from
the absence of files. But, in the end, it really depends on whether the
user believes you when you say it is temporary.

*************
There is nothing wrong with me that could
not be cured by a capful of Drano.

H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
"Model-Based Translation: The Next Step in Agile Development". Email
info@pathfindermda.com for your copy.
Pathfinder is hiring:
http://www.pathfindermda.com/about_us/careers_pos3.php.
(888)OOA-PATH

Phlip

2006-09-14, 7:00 pm

H. S. Lahman wrote:

> I would google for this. There are a number of GUI monitoring tools on
> the market that can be inserted between the OS Window Manger and the
> application to provide logging of the Window Manager messages.

Has anyone ever told their customers to simply leave TechSmith's Snagit
running in record mode while they use their interfaces? That would simply
collect a movie of screen captures leading up to the crashie.

(The OP is also advised to improve their coding practices, beginning with
automated testing...)

--
Phlip
[url]http://www.greencheese.us/Z

Land[/url] <-- NOT a blog!!!

sheldon.nicholl@state.or.us

2006-10-02, 7:02 pm

Are there any other video options? Thanks

Phlip wrote:
> H. S. Lahman wrote:
>
>
> Has anyone ever told their customers to simply leave TechSmith's Snagit
> running in record mode while they use their interfaces? That would simply
> collect a movie of screen captures leading up to the crashie.
>
> (The OP is also advised to improve their coding practices, beginning with
> automated testing...)
>
> --
> Phlip
> [url]http://www.greencheese.us/Z

Land[/url] <-- NOT a blog!!!

Sponsored Links