File Descriptors With High Values

I'm trying to get to the bottom of a problem that plagues my
application only on the GNU/Linux platform. The program is a
multithreaded http stress tester (siege). In the current beta, I am
running out of file descriptors. I've added debugging statements and
"-1" checks and I'm nearly positive that all open sockets are being
closed. In some runs the problem appears and in other runs it doesn't.
I added a debugging statement to print the socket file descriptor's
value to screen. In runs where the problem appears (too many open
files), most of the sockets are valued over 100 and on runs where it
doesn't appear, they are valued in single digits. I can tell the second
the run begins whether or not the problem will occur as the very first
descriptors be printed with those high values.

Any thoughts?

Jeff

Jeff wrote:
> I'm trying to get to the bottom of a problem that plagues my
> application only on the GNU/Linux platform. The program is a
> multithreaded http stress tester (siege). In the current beta, I am
> running out of file descriptors. I've added debugging statements and
> "-1" checks and I'm nearly positive that all open sockets are being
> closed.

Nearly positive?  Have you tried running lsof to see how many (and for
what purpose) files/sockets/etc your process has open?  That may just
tell
you the answer.

> In some runs the problem appears and in other runs it doesn't.
> I added a debugging statement to print the socket file descriptor's
> value to screen. In runs where the problem appears (too many open
> files), most of the sockets are valued over 100 and on runs where it
> doesn't appear, they are valued in single digits. I can tell the
second
> the run begins whether or not the problem will occur as the very
first
> descriptors be printed with those high values.
>
> Any thoughts?
>
> Jeff

I've not seen this.  Usually file descriptors start at the bottom
(after stderr) and fill in holes after old ones are closed.  Randomly
issuing higher numbered file descriptors when lower ones are available
wouldn't normally be done, because, for example, if you want to do
fd_set manipulations on a fd numbered higher than FD_SETSIZE you are
out of luck.

-David

lndresn...@gmail.com wrote:
> Jeff wrote: 
and 
>
> Nearly positive?  Have you tried running lsof to see how many (and
for
> what purpose) files/sockets/etc your process has open?  That may just
> tell you the answer.
>

I understand this may be a chicken and an egg so I'm reluctant to claim
that I am indeed positive. When the program starts with low-numbered
file descriptors, then yes, every socket gets closed. When the program
begins with high numbered file descriptors, then there is a chance a
couple threads may hang in close.

 
it 
> second 
> first 
>
> I've not seen this.  Usually file descriptors start at the bottom
> (after stderr) and fill in holes after old ones are closed.  Randomly
> issuing higher numbered file descriptors when lower ones are
available
> wouldn't normally be done, because, for example, if you want to do
> fd_set manipulations on a fd numbered higher than FD_SETSIZE you are
> out of luck.

"Good" runs display this behavior. "Bad" runs do not. This code works
like a charm on solaris and HPUX. Only GNU/Linux gives me a problem.

Jeff

lndresnick@gmail.com writes:

> I've not seen this.  Usually file descriptors start at the bottom
> (after stderr) and fill in holes after old ones are closed.  Randomly

This is required by some standard, I forget which one.

--
Måns Rullgård
mru@inprovide.com

Jeff wrote:
> lndresn...@gmail.com wrote:
> 
>
> and
> 
>
> for
> 
>
>
> I understand this may be a chicken and an egg so I'm reluctant to claim
> that I am indeed positive. When the program starts with low-numbered
> file descriptors, then yes, every socket gets closed. When the program
> begins with high numbered file descriptors, then there is a chance a
> couple threads may hang in close.
>
>
> 
>
> it
> 
>
> available
> 
>
>
> "Good" runs display this behavior. "Bad" runs do not. This code works
> like a charm on solaris and HPUX. Only GNU/Linux gives me a problem.
>
> Jeff
>

Just what is your user limit on file descriptors?  Frequently, you can
up the soft limit to the hard limit using getrlimit and setrlimit.
However, if you're talking about FILE * streams, then on many systems,
the upper limit for a file descriptor number is 255.

In the past when I was confronted with leaking file descriptors, I
would put a printf near every open and close detailing the place
in the code, and the file descriptor used.  This usually does
not take long to find the leak.  For FILE * streams, you can
use fileno to extract the file descriptor from the FILE *.

--

Fletcher Glenn

"Jeff" <joesiege@gmail.com> wrote in message
news:1113940426.919243.78170@g14g2000cwa.googlegroups.com...
> I'm trying to get to the bottom of a problem that plagues my
> application only on the GNU/Linux platform. The program is a
> multithreaded http stress tester (siege). In the current beta, I am
> running out of file descriptors. I've added debugging statements and
> "-1" checks and I'm nearly positive that all open sockets are being
> closed. In some runs the problem appears and in other runs it doesn't.
> I added a debugging statement to print the socket file descriptor's
> value to screen. In runs where the problem appears (too many open
> files), most of the sockets are valued over 100 and on runs where it
> doesn't appear, they are valued in single digits. I can tell the second
> the run begins whether or not the problem will occur as the very first
> descriptors be printed with those high values.

The other suggestions are good ones, but here's another possibility. If
your program is started in some odd way, it may inherit a bunch of file
descriptors it has no interest in. You can use the following code early in
main:

int i;
for(i=getdtablesize(); i>2; i++)
close(i);

This should close any inherited file descriptors that you don't want.
Ignore errors from 'close'.

DS

"David Schwartz" <davids@webmaster.com> writes:


>    The other suggestions are good ones, but here's another possibility. If
>your program is started in some odd way, it may inherit a bunch of file
>descriptors it has no interest in. You can use the following code early in
>main:

>int i;
>for(i=getdtablesize(); i>2; i++)
> close(i);

>    This should close any inherited file descriptors that you don't want.
>Ignore errors from 'close'.


But note that om some operating systems this may result in many thousands
of close() calls.

(That's why Solaris has "closefrom()" which more efficiently closes
all unwanted fds; it does so by reading /proc/self/fd/* and only closing
the fds listed there; but then again, Solaris is probably one of the
few Unix like OSes where the number of process file descriptors is
virtually unlimited)

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Casper H.S. Dik <Casper.Dik@Sun.COM> writes:

> "David Schwartz" <davids@webmaster.com> writes:
> 
> 

That should probably be i--, unless you really want to close
everything that can't be a file descriptor.
 
> 
>
> But note that om some operating systems this may result in many thousands
> of close() calls.

A few thousand calls is probably nothing to worry about, but if you
start reaching hundreds of thousands, or millions, there might be an
issue.

--
Måns Rullgård
mru@inprovide.com

David Schwartz wrote:
>
>     The other suggestions are good ones, but here's another
possibility. If
> your program is started in some odd way, it may inherit a bunch of
file
> descriptors it has no interest in. You can use the following code
early in
> main:
>
> int i;
> for(i=getdtablesize(); i>2; i++)
>  close(i);
>
>     This should close any inherited file descriptors that you don't
want.
> Ignore errors from 'close'.
>
>     DS

I tried this. Every single close returns -1 with errno set to EBADF

Jeff

Jeff wrote:

> Any thoughts?

0) just a descriptor leak. maybe you are closing the wrong descriptors ?

1) netstat -a | grep WAIT

2) maybe a router/firewall with limited resources ?

HTH,
AvK