Unix Programming - How to debug the linker-loader?

Author

How to debug the linker-loader?

Ramon F Herrera

2007-03-16, 10:04 pm

I write C programs which are later called from Perl CGI scripts. When
I run the C program from the shell, it runs properly. Then when I run
the enclosing Perl script, it also runs properly. However, sometimes
when the program is being executed from the web, sometimes it fails to
run.

A couple of solutions to the above problem that have worked for me
are:

(1) Link the C program statically
(2) Add a line such as:
/home/oracle/lib
to the file /etc/ld.so.cache

At this time, however, I would like to investigate the issue more
closely. Open the famous Linux hood, if you will.

How can I monitor the workings of the linker-loader when the program
in question is executed through the web?

A related question: is there any way at all to attach the gdb debugger
to one such program, which is not started by the shell? I have read
some comments that technique. That feat sounds like major wizardry.

TIA,

-Ramon F Herrera

Sam

2007-03-16, 10:04 pm

Bin Chen

2007-03-17, 4:15 am

On Mar 17, 10:41 am, Sam <s...@email-scan.com> wrote:
> Ramon F Herrera writes:
>
>
>
>
>
>
> The general approach is to wrap your program by a shell script.
>
> #! /bin/sh
>
> strace -o /tmp/strace.log /path/to/program
>
> Make this executable, and run it via http.
>
>
> Not really. Just put a sleep(15) call at the first order of business, in
> main(). Then, you have 15 seconds to run ps -ef | grep programname, grab
> its PID, and attach a debugger to it: gdb /path/to/program $PID. This also
> works with gdb wrappers, such as ddd.
Make it more automatic. Add a stub in you main():

#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
int pid;

pid = fork();
if (!pid) {
/*
* child
*/
char buf[128];
snprintf(buf, 128, "gdbserver :9988 --attach %d", getppid());
system(buf);
}

while (1)
sleep(5);
exit(0);
}

>
> After attaching the debugger, 's' to step out of sleep().
>
> On older Linux kernels, attaching a debugger in a middle of sleep()
> terminates the sleep immediately. With current kernel, 's' continues to
> sleep for whatever time remains, for sleeping; so adjust the sleep interval
> to match your typing speed and your mental space. Guess too short and
> you'll miss it. Guess too long, and you'll be sitting there, staring at the
> screen, waiting until the process wakes up.

Bin Chen

2007-03-17, 4:15 am

On Mar 17, 10:03 am, "Ramon F Herrera" <r...@conexus.net> wrote:
> I write C programs which are later called from Perl CGI scripts. When
> I run the C program from the shell, it runs properly. Then when I run
> the enclosing Perl script, it also runs properly. However, sometimes
> when the program is being executed from the web, sometimes it fails to
> run.
>
> A couple of solutions to the above problem that have worked for me
> are:
>
> (1) Link the C program statically
> (2) Add a line such as:
> /home/oracle/lib
> to the file /etc/ld.so.cache
>
> At this time, however, I would like to investigate the issue more
> closely. Open the famous Linux hood, if you will.
>
> How can I monitor the workings of the linker-loader when the program
> in question is executed through the web?
>
> A related question: is there any way at all to attach the gdb debugger
> to one such program, which is not started by the shell? I have read
> some comments that technique. That feat sounds like major wizardry.

Honestly speaking, your problem seems can't be debug this way to find
out the problem. When kernel executes a executable, it first load the
loader into the memory, and after the loader done the necessary all
the 'dynamic' task, it will transfer the control to the main(), which
is the very first place that you can put your modification in. But
your problem mostly probably resides in the loader part before main().
So debugger may not survive.

Paul Pluzhnikov

2007-03-17, 4:15 am

Sam <sam@email-scan.com> writes:

> Ramon F Herrera writes:
>

Fails to run how?

Your subject line implies that it fails due to some ld-linux.so.2
error, but without explicit symptom description it's hard to give
you meaningful advice.
[color=darkred]

This probably just hides the bug. Your program may still fail,
or it may fail when you move it to updated system, or may start
failing when you make arbitratily small changes to it.
[color=darkred]

Again this likely merely masks the problem.
[color=darkred]

The fact that it's executed 'through the web' is irrelevant.
You can wrap it in a shell script, or a perl script, or another
executable, and control it just the same way as if it was executed
from command line. The only "limitation" is that its output goes
to a file (/var/log/httpd/error_log).
[color=darkred]
> The general approach is to wrap your program by a shell script.
>
> #! /bin/sh
> strace -o /tmp/strace.log /path/to/program

There are several problems with this advice:
- command-line arguments to the "wrapper" are ignored.
Better add "$@" (do include quotes) after /path/to/program

- strace may or may not be the right tool for debugging this.

- if the problem really is related to dynamic loader, this may
provide more useful info than strace:
LD_DEBUG=files /path/to/program "$@"

- in a web environment, strace.log will likely be quickly
overwritten, so better make it semi-unique with -o /tmp/strace.$$.log

- also, it's better to do 'exec strace -o ...'

Sure.
[color=darkred]

It's not. All you do is figure out pid of the misbehaving process, then

gdb /path/to/program <pid>

and presto -- you are debugging it.
[color=darkred]
> Not really.

Yes, really. Since you explain *how* to do it below, your "No"
answer makes no sense.

> Just put a sleep(15) call at the first order of business,
> in main().

OP needs to do that only if he needs to debug a problem that happens
"close" to main(). He stated no such requirement.

> Then, you have 15 seconds to run ps -ef | grep
> programname, grab its PID, and attach a debugger to it: gdb
> /path/to/program $PID. This also works with gdb wrappers, such as ddd.

It is better to use pgrep instead of 'ps -ef'.

A techhique that allows "unlimited" wait time:

int main(int argc, char *argv[])
{
if (getenv("WAIT_FOR_DEBUGGER")) {
int wait = 1;
fprintf(stderr,
"%s: pid: %d is waiting for debugger to attach\n"
"gdb %s %d\n",
argv[0], getpid(), argv[0], getpid());
while (wait) { sleep(1); } // debugger will attach here
}
...

This way the program tells you what pid you need to attach to,
and you can take your time doing so.

Once attached, "set var wait = 0" and "continue".

Just don't forget to unset WAIT_FOR_DEBUGGER when you are done.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Sponsored Links