identical copies of JSP site behave differently

Hi,

I have a web application, coded in JSP/Servlets, which is running on our
production Redhat box. I also have a testing environment setup on a
different server (again Redhat), which is essentially an exact copy of
the production.

Both instances access the same database (physically located on the
production server).

The problem is, we have some bugs in the application, but they are only
reproduce able on the production server and not the qa server.

Environment:
The JDK is j2sdk1.4.1_07.
Both servers are running virtually the same OS. Only differences are:
Live: dual proc with SMP kernel, packages compiled by RH
qa: single proc, packages compiled by CentOS team
(What am I missing here?)

Thanks

On Wed, 20 Apr 2005, Justin wrote:

> Hi,
>
> I have a web application, coded in JSP/Servlets, which is running on our
> production Redhat box. I also have a testing environment setup on a
> different server (again Redhat), which is essentially an exact copy of
> the production.
>
> Both instances access the same database (physically located on the
> production server).
>
> The problem is, we have some bugs in the application, but they are only
> reproduce able on the production server and not the qa server.
>
> Environment:
> The JDK is j2sdk1.4.1_07.
> Both servers are running virtually the same OS. Only differences are:
> Live: dual proc with SMP kernel, packages compiled by RH
> qa: single proc, packages compiled by CentOS team
> (What am I missing here?)

Dual processor means that you can have two threads, literally, running at
the same time. Maybe the problem is related to that. The JSP is compiling
into servlet code that needs to be sync'd but it is not.

Another possibility is the compiler. Two different compilers could make
different code. Since this is a second difference I would see if you can
compile using the CentOS team and run on the production environment. As a
double check you can also try compiling by RH and run on the qa
environment.

This will mean four combinations:

fail: dual, RH
pass: single, CentOS
????: dual, CentOS
????: single, RH

If the dual, CentOS fails then the problem is related to the dual
processor. If the single, RH fails then the problem is related to the
compiler. If the results are inconclusive then maybe there is a third
factor and you don't realize it.

Bottom line, try combinations until you narrow it down to one factor.

--
Send e-mail to: darrell dot grainger at utoronto dot ca

Justin wrote:

> Hi,
>
> I have a web application, coded in JSP/Servlets, which is running on our
> production Redhat box. I also have a testing environment setup on a
> different server (again Redhat), which is essentially an exact copy of
> the production.
>
> Both instances access the same database (physically located on the
> production server).
>
> The problem is, we have some bugs in the application, but they are only
> reproduce able on the production server and not the qa server.
>
> Environment:
> The JDK is j2sdk1.4.1_07.
> Both servers are running virtually the same OS. Only differences are:
> Live: dual proc with SMP kernel, packages compiled by RH
> qa: single proc, packages compiled by CentOS team
> (What am I missing here?)
>
> Thanks

Given these symptoms, I would do a synchronization design
review.

A single processor system runs threads in parallel only by
time slicing them. In many applications each thread runs for
only a short time between events that voluntarily give up
the processor such as I/O and wait. In that case, most
switches will happen with the thread that is giving up the
processor in a stable state at a place where it e.g. waits
for I/O. Threads could execute code with missing
synchronization with relatively low risk of symptoms, as
long as it does not wait for I/O in the middle of doing it.

In effect, possession of the one-and-only processor acts as
a sort of soft lock, excluding other threads until this
thread reaches a place where it gives up the processor.

Now consider a dual processor. At any time, two threads
really are running in parallel. Any missing synchronization
has a much higher probability of turning into visible symptoms.

As a temporary workaround and diagnostic check, while you
are debugging it, see if the OS on the production server
allows binding of your job to one processor. If you have a
synchronization problem it will go away. If the problem is
due to some other difference in the operating environments,
it won't. Of course, you may take a performance hit, and it
is not a solid fix. At best, it will drastically reduce the
frequency of the bug.

On the test system, try constructing a workload with some
higher priority jobs that do very small amounts of compute
interleaved with waiting. The objective is to use a high
priority thread becoming runnable to take the processor away
from the web application at arbitrary times, not just when
it gives it up.

Patricia