Home > Archive > Unix Programming > April 2005 > File Descriptors With High Values
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
File Descriptors With High Values
|
|
|
| I'm trying to get to the bottom of a problem that plagues my
application only on the GNU/Linux platform. The program is a
multithreaded http stress tester (siege). In the current beta, I am
running out of file descriptors. I've added debugging statements and
"-1" checks and I'm nearly positive that all open sockets are being
closed. In some runs the problem appears and in other runs it doesn't.
I added a debugging statement to print the socket file descriptor's
value to screen. In runs where the problem appears (too many open
files), most of the sockets are valued over 100 and on runs where it
doesn't appear, they are valued in single digits. I can tell the second
the run begins whether or not the problem will occur as the very first
descriptors be printed with those high values.
Any thoughts?
Jeff
| |
| lndresnick@gmail.com 2005-04-19, 8:57 pm |
|
Jeff wrote:
> I'm trying to get to the bottom of a problem that plagues my
> application only on the GNU/Linux platform. The program is a
> multithreaded http stress tester (siege). In the current beta, I am
> running out of file descriptors. I've added debugging statements and
> "-1" checks and I'm nearly positive that all open sockets are being
> closed.
Nearly positive? Have you tried running lsof to see how many (and for
what purpose) files/sockets/etc your process has open? That may just
tell
you the answer.
> In some runs the problem appears and in other runs it doesn't.
> I added a debugging statement to print the socket file descriptor's
> value to screen. In runs where the problem appears (too many open
> files), most of the sockets are valued over 100 and on runs where it
> doesn't appear, they are valued in single digits. I can tell the
second
> the run begins whether or not the problem will occur as the very
first
> descriptors be printed with those high values.
>
> Any thoughts?
>
> Jeff
I've not seen this. Usually file descriptors start at the bottom
(after stderr) and fill in holes after old ones are closed. Randomly
issuing higher numbered file descriptors when lower ones are available
wouldn't normally be done, because, for example, if you want to do
fd_set manipulations on a fd numbered higher than FD_SETSIZE you are
out of luck.
-David
| |
|
|
lndresn...@gmail.com wrote:
> Jeff wrote:
and[color=darkred]
>
> Nearly positive? Have you tried running lsof to see how many (and
for
> what purpose) files/sockets/etc your process has open? That may just
> tell you the answer.
>
I understand this may be a chicken and an egg so I'm reluctant to claim
that I am indeed positive. When the program starts with low-numbered
file descriptors, then yes, every socket gets closed. When the program
begins with high numbered file descriptors, then there is a chance a
couple threads may hang in close.
it[color=darkred]
> second
> first
>
> I've not seen this. Usually file descriptors start at the bottom
> (after stderr) and fill in holes after old ones are closed. Randomly
> issuing higher numbered file descriptors when lower ones are
available
> wouldn't normally be done, because, for example, if you want to do
> fd_set manipulations on a fd numbered higher than FD_SETSIZE you are
> out of luck.
"Good" runs display this behavior. "Bad" runs do not. This code works
like a charm on solaris and HPUX. Only GNU/Linux gives me a problem.
Jeff
| |
| Måns Rullgård 2005-04-19, 8:57 pm |
| lndresnick@gmail.com writes:
> I've not seen this. Usually file descriptors start at the bottom
> (after stderr) and fill in holes after old ones are closed. Randomly
This is required by some standard, I forget which one.
--
Måns Rullgård
mru@inprovide.com
| |
| Fletcher Glenn 2005-04-19, 8:57 pm |
| Jeff wrote:
> lndresn...@gmail.com wrote:
>
>
> and
>
>
> for
>
>
>
> I understand this may be a chicken and an egg so I'm reluctant to claim
> that I am indeed positive. When the program starts with low-numbered
> file descriptors, then yes, every socket gets closed. When the program
> begins with high numbered file descriptors, then there is a chance a
> couple threads may hang in close.
>
>
>
>
> it
>
>
> available
>
>
>
> "Good" runs display this behavior. "Bad" runs do not. This code works
> like a charm on solaris and HPUX. Only GNU/Linux gives me a problem.
>
> Jeff
>
Just what is your user limit on file descriptors? Frequently, you can
up the soft limit to the hard limit using getrlimit and setrlimit.
However, if you're talking about FILE * streams, then on many systems,
the upper limit for a file descriptor number is 255.
In the past when I was confronted with leaking file descriptors, I
would put a printf near every open and close detailing the place
in the code, and the file descriptor used. This usually does
not take long to find the leak. For FILE * streams, you can
use fileno to extract the file descriptor from the FILE *.
--
Fletcher Glenn
| |
| David Schwartz 2005-04-20, 3:59 am |
|
"Jeff" <joesiege@gmail.com> wrote in message
news:1113940426.919243.78170@g14g2000cwa.googlegroups.com...
> I'm trying to get to the bottom of a problem that plagues my
> application only on the GNU/Linux platform. The program is a
> multithreaded http stress tester (siege). In the current beta, I am
> running out of file descriptors. I've added debugging statements and
> "-1" checks and I'm nearly positive that all open sockets are being
> closed. In some runs the problem appears and in other runs it doesn't.
> I added a debugging statement to print the socket file descriptor's
> value to screen. In runs where the problem appears (too many open
> files), most of the sockets are valued over 100 and on runs where it
> doesn't appear, they are valued in single digits. I can tell the second
> the run begins whether or not the problem will occur as the very first
> descriptors be printed with those high values.
The other suggestions are good ones, but here's another possibility. If
your program is started in some odd way, it may inherit a bunch of file
descriptors it has no interest in. You can use the following code early in
main:
int i;
for(i=getdtablesize(); i>2; i++)
close(i);
This should close any inherited file descriptors that you don't want.
Ignore errors from 'close'.
DS
| |
| Casper H.S. Dik 2005-04-20, 8:58 am |
| "David Schwartz" <davids@webmaster.com> writes:
> The other suggestions are good ones, but here's another possibility. If
>your program is started in some odd way, it may inherit a bunch of file
>descriptors it has no interest in. You can use the following code early in
>main:
>int i;
>for(i=getdtablesize(); i>2; i++)
> close(i);
> This should close any inherited file descriptors that you don't want.
>Ignore errors from 'close'.
But note that om some operating systems this may result in many thousands
of close() calls.
(That's why Solaris has "closefrom()" which more efficiently closes
all unwanted fds; it does so by reading /proc/self/fd/* and only closing
the fds listed there; but then again, Solaris is probably one of the
few Unix like OSes where the number of process file descriptors is
virtually unlimited)
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
| |
| Måns Rullgård 2005-04-20, 8:58 am |
| Casper H.S. Dik <Casper.Dik@Sun.COM> writes:
> "David Schwartz" <davids@webmaster.com> writes:
>
>
That should probably be i--, unless you really want to close
everything that can't be a file descriptor.
[color=darkred]
>
>
> But note that om some operating systems this may result in many thousands
> of close() calls.
A few thousand calls is probably nothing to worry about, but if you
start reaching hundreds of thousands, or millions, there might be an
issue.
--
Måns Rullgård
mru@inprovide.com
| |
|
|
David Schwartz wrote:
>
> The other suggestions are good ones, but here's another
possibility. If
> your program is started in some odd way, it may inherit a bunch of
file
> descriptors it has no interest in. You can use the following code
early in
> main:
>
> int i;
> for(i=getdtablesize(); i>2; i++)
> close(i);
>
> This should close any inherited file descriptors that you don't
want.
> Ignore errors from 'close'.
>
> DS
I tried this. Every single close returns -1 with errno set to EBADF
Jeff
| |
|
| Jeff wrote:
> Any thoughts?
0) just a descriptor leak. maybe you are closing the wrong descriptors ?
1) netstat -a | grep WAIT
2) maybe a router/firewall with limited resources ?
HTH,
AvK
| |
| lndresnick@gmail.com 2005-04-20, 3:59 pm |
|
Jeff wrote:
> David Schwartz wrote:
> possibility. If
> file
> early in
> want.
>
> I tried this. Every single close returns -1 with errno set to EBADF
>
> Jeff
Did you see the reply from M=E5ns Rullg=E5rd suggesting that i-- was what
was wanted, i.e. close all file descriptors from the highest one you
have down
to 2 (stderr).
I still say, run lsof on a bad run and see what the descriptors in use
actually are. Have you tried? What do you see? The idea that you've
inherited descriptors from a parent (via fork say) seems attractive.
-David
| |
|
| lndresn...@gmail.com wrote:
> Jeff wrote:
of[color=darkred]
don't[color=darkred]
>
> Did you see the reply from M=E5ns Rullg=E5rd suggesting that i-- was
what
> was wanted, i.e. close all file descriptors from the highest one you
> have down
> to 2 (stderr).
>
> I still say, run lsof on a bad run and see what the descriptors in
use
> actually are. Have you tried? What do you see? The idea that
you've
> inherited descriptors from a parent (via fork say) seems attractive.
>
> -David
Yes. I corrected the code. On a bad run, I get a thread/threads stuck
in CLOSE. Here's an example from lsof
19283 jeff cwd DIR 3,3 4096 341003
/home/jeff/src/siege/src
siege 19283 jeff rtd DIR 3,1 1024 2 /
siege 19283 jeff txt REG 3,3 2514854 341739
/home/jeff/src/siege/src/siege
siege 19283 jeff mem REG 3,1 94543 30201
/lib/ld-2.2.4.so
siege 19283 jeff mem REG 3,1 106843 30216
/lib/libpthread.so.0
siege 19283 jeff mem REG 3,1 1384168 30203
/lib/libc.so.6
siege 19283 jeff mem REG 3,1 44166 30212
/lib/libnss_files.so.2
siege 19283 jeff mem REG 3,1 16408 30211
/lib/libnss_dns.so.2
siege 19283 jeff mem REG 3,1 66718 30217
/lib/libresolv.so.2
siege 19283 jeff 0u CHR 136,3 5
/dev/pts/3
siege 19283 jeff 1u CHR 136,3 5
/dev/pts/3
siege 19283 jeff 2u CHR 136,3 5
/dev/pts/3
siege 19283 jeff 3r FIFO 0,0 2401650 pipe
siege 19283 jeff 4w FIFO 0,0 2401650 pipe
siege 19283 jeff 10u sock 0,0 2402551 can't
identify protocol
siege 19283 jeff 12u sock 0,0 2402552 can't
identify protocol
siege 19283 jeff 13u sock 0,0 2401659 can't
identify protocol
siege 19283 jeff 44u sock 0,0 2405484 can't
identify protocol
siege 19283 jeff 49u sock 0,0 2401695 can't
identify protocol
siege 19283 jeff 58u sock 0,0 2405171 can't
identify protocol
siege 19283 jeff 74u sock 0,0 2405165 can't
identify protocol
siege 19283 jeff 84u sock 0,0 2403054 can't
identify protocol
siege 19283 jeff 85u sock 0,0 2404246 can't
identify protocol
siege 19283 jeff 86u IPv4 2405383 TCP
eos.joedog.org:3720->ben.joedog.org:http (CLOSE)
siege 19283 jeff 115u sock 0,0 2403418 can't
identify protocol
siege 19283 jeff 128u sock 0,0 2405472 can't
identify protocol
siege 19283 jeff 141u sock 0,0 2403347 can't
identify protocol
siege 19283 jeff 176u sock 0,0 2405174 can't
identify protocol
siege 19283 jeff 210u sock 0,0 2404704 can't
identify protocol
siege 19283 jeff 261u sock 0,0 2401907 can't
identify protocol
siege 19283 jeff 263u sock 0,0 2401909 can't
identify protocol
siege 19283 jeff 265u sock 0,0 2401911 can't
identify protocol
siege 19283 jeff 267u sock 0,0 2401913 can't
identify protocol
siege 19283 jeff 269u sock 0,0 2401915 can't
identify protocol
siege 19283 jeff 271u sock 0,0 2401917 can't
identify protocol
siege 19283 jeff 273u sock 0,0 2401919 can't
identify protocol
siege 19283 jeff 275u sock 0,0 2401921 can't
identify protocol
siege 19283 jeff 277u sock 0,0 2401923 can't
identify protocol
siege 19283 jeff 279u sock 0,0 2401925 can't
identify protocol
siege 19283 jeff 283u sock 0,0 2401929 can't
identify protocol
siege 19283 jeff 285u sock 0,0 2401931 can't
identify protocol
siege 19283 jeff 287u sock 0,0 2401933 can't
identify protocol
siege 19283 jeff 289u sock 0,0 2401935 can't
identify protocol
siege 19283 jeff 291u sock 0,0 2401937 can't
identify protocol
siege 19283 jeff 293u sock 0,0 2401939 can't
identify protocol
siege 19283 jeff 295u sock 0,0 2401941 can't
identify protocol
siege 19283 jeff 297u sock 0,0 2401943 can't
identify protocol
siege 19283 jeff 299u sock 0,0 2401945 can't
identify protocol
siege 19283 jeff 303u sock 0,0 2401949 can't
identify protocol
siege 19283 jeff 305u sock 0,0 2401951 can't
identify protocol
siege 19283 jeff 307u sock 0,0 2401953 can't
identify protocol
siege 19283 jeff 309u sock 0,0 2401955 can't
identify protocol
siege 19283 jeff 311u sock 0,0 2401957 can't
identify protocol
siege 19283 jeff 315u sock 0,0 2401961 can't
identify protocol
siege 19283 jeff 317u sock 0,0 2401963 can't
identify protocol
siege 19283 jeff 319u sock 0,0 2401965 can't
identify protocol
siege 19283 jeff 321u sock 0,0 2401967 can't
identify protocol
siege 19283 jeff 323u sock 0,0 2401969 can't
identify protocol
siege 19283 jeff 325u sock 0,0 2401971 can't
identify protocol
siege 19283 jeff 327u sock 0,0 2401973 can't
identify protocol
siege 19283 jeff 329u sock 0,0 2401975 can't
identify protocol
siege 19283 jeff 331u sock 0,0 2401977 can't
identify protocol
siege 19283 jeff 333u sock 0,0 2401979 can't
identify protocol
siege 19283 jeff 335u sock 0,0 2401981 can't
identify protocol
siege 19283 jeff 337u sock 0,0 2401983 can't
identify protocol
siege 19283 jeff 341u sock 0,0 2401987 can't
identify protocol
siege 19283 jeff 342u sock 0,0 2405460 can't
identify protocol
siege 19283 jeff 343u sock 0,0 2401989 can't
identify protocol
siege 19283 jeff 345u sock 0,0 2401991 can't
identify protocol
siege 19283 jeff 346u sock 0,0 2405468 can't
identify protocol
siege 19283 jeff 347u sock 0,0 2401993 can't
identify protocol
siege 19283 jeff 349u sock 0,0 2401995 can't
identify protocol
siege 19283 jeff 351u sock 0,0 2401997 can't
identify protocol
siege 19283 jeff 353u sock 0,0 2401999 can't
identify protocol
siege 19283 jeff 355u sock 0,0 2402001 can't
identify protocol
siege 19283 jeff 357u sock 0,0 2402003 can't
identify protocol
siege 19283 jeff 359u sock 0,0 2402005 can't
identify protocol
siege 19283 jeff 361u sock 0,0 2402007 can't
identify protocol
siege 19283 jeff 363u sock 0,0 2402009 can't
identify protocol
siege 19283 jeff 365u sock 0,0 2402011 can't
identify protocol
siege 19283 jeff 367u sock 0,0 2402013 can't
identify protocol
siege 19283 jeff 369u sock 0,0 2402015 can't
identify protocol
siege 19283 jeff 371u sock 0,0 2402017 can't
identify protocol
siege 19283 jeff 373u sock 0,0 2402019 can't
identify protocol
siege 19283 jeff 404u sock 0,0 2404706 can't
identify protocol
siege 19283 jeff 411u sock 0,0 2402057 can't
identify protocol
siege 19283 jeff 413u sock 0,0 2402059 can't
identify protocol
siege 19283 jeff 415u sock 0,0 2402061 can't
identify protocol
siege 19283 jeff 417u sock 0,0 2402063 can't
identify protocol
siege 19283 jeff 419u sock 0,0 2402065 can't
identify protocol
siege 19283 jeff 421u sock 0,0 2402067 can't
identify protocol
siege 19283 jeff 423u sock 0,0 2402069 can't
identify protocol
siege 19283 jeff 424u IPv4 2402801 UDP
eos.joedog.org:3550->ns.joedog.org:domain
siege 19283 jeff 425u sock 0,0 2402071 can't
identify protocol
siege 19283 jeff 427u sock 0,0 2402073 can't
identify protocol
siege 19283 jeff 429u sock 0,0 2402075 can't
identify protocol
siege 19283 jeff 433u sock 0,0 2405178 can't
identify protocol
siege 19283 jeff 457u sock 0,0 2405481 can't
identify protocol
siege 19283 jeff 497u sock 0,0 2402143 can't
identify protocol
siege 19283 jeff 502u IPv4 2402876 TCP
eos.joedog.org:2956->ben.joedog.org:http (CLOSE)
siege 19283 jeff 512u sock 0,0 2405162 can't
identify protocol
siege 19283 jeff 523u sock 0,0 2402169 can't
identify protocol
siege 19283 jeff 529u sock 0,0 2402175 can't
identify protocol
siege 19283 jeff 531u sock 0,0 2402177 can't
identify protocol
siege 19283 jeff 535u sock 0,0 2402181 can't
identify protocol
siege 19283 jeff 552u sock 0,0 2403021 can't
identify protocol
siege 19283 jeff 556u sock 0,0 2403023 can't
identify protocol
siege 19283 jeff 559u sock 0,0 2404229 can't
identify protocol
siege 19283 jeff 560u sock 0,0 2403025 can't
identify protocol
siege 19283 jeff 564u sock 0,0 2403027 can't
identify protocol
siege 19283 jeff 568u sock 0,0 2403029 can't
identify protocol
siege 19283 jeff 569u sock 0,0 2402215 can't
identify protocol
siege 19283 jeff 572u sock 0,0 2403031 can't
identify protocol
siege 19283 jeff 576u sock 0,0 2403033 can't
identify protocol
siege 19283 jeff 577u sock 0,0 2402223 can't
identify protocol
siege 19283 jeff 579u sock 0,0 2402225 can't
identify protocol
siege 19283 jeff 581u sock 0,0 2402227 can't
identify protocol
siege 19283 jeff 583u sock 0,0 2402229 can't
identify protocol
siege 19283 jeff 585u sock 0,0 2402231 can't
identify protocol
siege 19283 jeff 586u sock 0,0 2403038 can't
identify protocol
siege 19283 jeff 587u sock 0,0 2402233 can't
identify protocol
siege 19283 jeff 589u sock 0,0 2402235 can't
identify protocol
siege 19283 jeff 590u sock 0,0 2403040 can't
identify protocol
siege 19283 jeff 591u sock 0,0 2402237 can't
identify protocol
siege 19283 jeff 593u sock 0,0 2402239 can't
identify protocol
siege 19283 jeff 594u sock 0,0 2403042 can't
identify protocol
siege 19283 jeff 597u sock 0,0 2402243 can't
identify protocol
siege 19283 jeff 598u sock 0,0 2403044 can't
identify protocol
siege 19283 jeff 599u sock 0,0 2402245 can't
identify protocol
siege 19283 jeff 601u sock 0,0 2402247 can't
identify protocol
siege 19283 jeff 602u sock 0,0 2403046 can't
identify protocol
siege 19283 jeff 603u sock 0,0 2402249 can't
identify protocol
siege 19283 jeff 605u sock 0,0 2403048 can't
identify protocol
siege 19283 jeff 607u sock 0,0 2403050 can't
identify protocol
siege 19283 jeff 609u sock 0,0 2403052 can't
identify protocol
siege 19283 jeff 612u sock 0,0 2403056 can't
identify protocol
siege 19283 jeff 614u sock 0,0 2403058 can't
identify protocol
siege 19283 jeff 645u sock 0,0 2403104 can't
identify protocol
siege 19283 jeff 647u sock 0,0 2403105 can't
identify protocol
siege 19283 jeff 648u sock 0,0 2403106 can't
identify protocol
siege 19283 jeff 649u sock 0,0 2403107 can't
identify protocol
siege 19283 jeff 650u sock 0,0 2403108 can't
identify protocol
siege 19283 jeff 654u sock 0,0 2403112 can't
identify protocol
siege 19283 jeff 655u sock 0,0 2403113 can't
identify protocol
siege 19283 jeff 656u sock 0,0 2403114 can't
identify protocol
siege 19283 jeff 657u sock 0,0 2403115 can't
identify protocol
siege 19283 jeff 658u sock 0,0 2403116 can't
identify protocol
siege 19283 jeff 659u sock 0,0 2403117 can't
identify protocol
siege 19283 jeff 662u sock 0,0 2403120 can't
identify protocol
siege 19283 jeff 663u sock 0,0 2403121 can't
identify protocol
siege 19283 jeff 664u sock 0,0 2403122 can't
identify protocol
siege 19283 jeff 665u sock 0,0 2403123 can't
identify protocol
siege 19283 jeff 666u sock 0,0 2403124 can't
identify protocol
siege 19283 jeff 667u sock 0,0 2403125 can't
identify protocol
siege 19283 jeff 668u sock 0,0 2403126 can't
identify protocol
siege 19283 jeff 669u sock 0,0 2403127 can't
identify protocol
siege 19283 jeff 670u sock 0,0 2403128 can't
identify protocol
siege 19283 jeff 671u sock 0,0 2403129 can't
identify protocol
siege 19283 jeff 672u sock 0,0 2403130 can't
identify protocol
siege 19283 jeff 704u sock 0,0 2404213 can't
identify protocol
siege 19283 jeff 708u sock 0,0 2403166 can't
identify protocol
siege 19283 jeff 712u sock 0,0 2403171 can't
identify protocol
siege 19283 jeff 715u sock 0,0 2403174 can't
identify protocol
siege 19283 jeff 738u sock 0,0 2404234 can't
identify protocol
siege 19283 jeff 747u sock 0,0 2403210 can't
identify protocol
siege 19283 jeff 767u sock 0,0 2403230 can't
identify protocol
siege 19283 jeff 783u sock 0,0 2403245 can't
identify protocol
siege 19283 jeff 784u sock 0,0 2403419 can't
identify protocol
siege 19283 jeff 785u sock 0,0 2403247 can't
identify protocol
siege 19283 jeff 786u sock 0,0 2403420 can't
identify protocol
siege 19283 jeff 787u sock 0,0 2403249 can't
identify protocol
siege 19283 jeff 789u sock 0,0 2403251 can't
identify protocol
siege 19283 jeff 796u sock 0,0 2403388 can't
identify protocol
siege 19283 jeff 798u sock 0,0 2403389 can't
identify protocol
siege 19283 jeff 800u sock 0,0 2403390 can't
identify protocol
siege 19283 jeff 802u sock 0,0 2403391 can't
identify protocol
siege 19283 jeff 803u sock 0,0 2403265 can't
identify protocol
siege 19283 jeff 804u sock 0,0 2403392 can't
identify protocol
siege 19283 jeff 805u sock 0,0 2403267 can't
identify protocol
siege 19283 jeff 806u sock 0,0 2403393 can't
identify protocol
siege 19283 jeff 807u sock 0,0 2403269 can't
identify protocol
siege 19283 jeff 808u sock 0,0 2403394 can't
identify protocol
siege 19283 jeff 809u sock 0,0 2403271 can't
identify protocol
siege 19283 jeff 810u sock 0,0 2403395 can't
identify protocol
siege 19283 jeff 811u sock 0,0 2403273 can't
identify protocol
siege 19283 jeff 812u sock 0,0 2403396 can't
identify protocol
siege 19283 jeff 813u sock 0,0 2403275 can't
identify protocol
siege 19283 jeff 814u sock 0,0 2403397 can't
identify protocol
siege 19283 jeff 815u sock 0,0 2403277 can't
identify protocol
siege 19283 jeff 817u sock 0,0 2403279 can't
identify protocol
siege 19283 jeff 818u sock 0,0 2403398 can't
identify protocol
siege 19283 jeff 819u sock 0,0 2403281 can't
identify protocol
siege 19283 jeff 820u sock 0,0 2403399 can't
identify protocol
siege 19283 jeff 821u sock 0,0 2403283 can't
identify protocol
siege 19283 jeff 822u sock 0,0 2403400 can't
identify protocol
siege 19283 jeff 823u sock 0,0 2403285 can't
identify protocol
siege 19283 jeff 824u sock 0,0 2403401 can't
identify protocol
siege 19283 jeff 825u sock 0,0 2403287 can't
identify protocol
siege 19283 jeff 826u sock 0,0 2403402 can't
identify protocol
siege 19283 jeff 827u sock 0,0 2403289 can't
identify protocol
siege 19283 jeff 828u sock 0,0 2403403 can't
identify protocol
siege 19283 jeff 834u sock 0,0 2403406 can't
identify protocol
siege 19283 jeff 835u sock 0,0 2403297 can't
identify protocol
siege 19283 jeff 836u sock 0,0 2403407 can't
identify protocol
siege 19283 jeff 837u sock 0,0 2403299 can't
identify protocol
siege 19283 jeff 838u sock 0,0 2403408 can't
identify protocol
siege 19283 jeff 840u sock 0,0 2403409 can't
identify protocol
siege 19283 jeff 842u sock 0,0 2403410 can't
identify protocol
siege 19283 jeff 844u sock 0,0 2403411 can't
identify protocol
siege 19283 jeff 846u sock 0,0 2403412 can't
identify protocol
siege 19283 jeff 848u sock 0,0 2403413 can't
identify protocol
siege 19283 jeff 850u sock 0,0 2403414 can't
identify protocol
siege 19283 jeff 852u sock 0,0 2403415 can't
identify protocol
siege 19283 jeff 853u sock 0,0 2403315 can't
identify protocol
siege 19283 jeff 861u sock 0,0 2403323 can't
identify protocol
siege 19283 jeff 863u sock 0,0 2403325 can't
identify protocol
siege 19283 jeff 872u sock 0,0 2405152 can't
identify protocol
siege 19283 jeff 873u sock 0,0 2403335 can't
identify protocol
siege 19283 jeff 875u sock 0,0 2403337 can't
identify protocol
siege 19283 jeff 877u sock 0,0 2403339 can't
identify protocol
siege 19283 jeff 878u sock 0,0 2405183 can't
identify protocol
siege 19283 jeff 879u sock 0,0 2403341 can't
identify protocol
siege 19283 jeff 881u sock 0,0 2403343 can't
identify protocol
siege 19283 jeff 883u sock 0,0 2403345 can't
identify protocol
siege 19283 jeff 886u sock 0,0 2403416 can't
identify protocol
siege 19283 jeff 887u sock 0,0 2403417 can't
identify protocol
siege 19283 jeff 1023u sock 0,0 2405455 can't
identify protocol
| |
| James Antill 2005-04-20, 8:58 pm |
| On Tue, 19 Apr 2005 12:53:46 -0700, Jeff wrote:
> I'm trying to get to the bottom of a problem that plagues my
> application only on the GNU/Linux platform. The program is a
> multithreaded http stress tester (siege).
You can't just say "the GNU/Linux" platform, I mean if you are using
Debian you're almost certainly using a non-POSIX threading library. But
that doesn't matter the problem is in your code.
As I'm interested in http stress testers I tried it out, here are the
problems I found ...
http.c
ssize_t http_read( CONN *C )
{
size_t bytes = 0;
size_t length = 0;
/* ... */
if( C->content.length > 0 ){
length = (C->content.length < MAXFILE)?C->content.length:MAXFILE;
do {
memset(body, 0, sizeof(body));
if(( n = socket_read(C, body, length)) == 0 )
break;
bytes += n;
length = (C->content.length - bytes < MAXFILE)?C->content.length-bytes:MAXFILE;
} while( bytes <= C->content.length );
....the while loop should be while ( bytes < c->content.length), to
simplify imagine the server says content-length == 1 ... then in the
above you will read that byte, and then _continue_ looping.
The next problem makes it worse...
sock.c
ssize_t socket_read(CONN *C, void *vbuf, size_t len)
{
size_t n;
/* ... */
n = len;
/* ... */
while(n > 0){
if(C->inbuffer < n){
lidos = read(C->sock, &C->buffer[C->inbuffer], sizeof(C->buffer)-C->inbuffer);
if( lidos < 0 ){
if(errno==EINTR || errno==EAGAIN)
lidos = 0;
/* shouldn't confuse EAGAIN with lidos == 0 ... they are _very_ different */
/* ... */
if(C->inbuffer >= n){
r = n;
} else {
r = C->inbuffer;
}
if(r == 0) break;
....so here in the 1 byte read() case, you are requesting a 0 byte read and
read is returning 0 ... but you are ignoring that because you already have
_some_ data in the incoming buffer.
This also means that if the remote machine decides to close() when
outputting a request you'll loop forever. So this needs to be fixed as
well.
--
James Antill -- james@and.org
http://www.and.org/vstr/httpd
| |
| James Antill 2005-04-20, 8:58 pm |
| On Wed, 20 Apr 2005 15:55:43 -0400, James Antill wrote:
> On Tue, 19 Apr 2005 12:53:46 -0700, Jeff wrote:
>
>
> You can't just say "the GNU/Linux" platform, I mean if you are using
> Debian you're almost certainly using a non-POSIX threading library. But
> that doesn't matter the problem is in your code.
> As I'm interested in http stress testers I tried it out, here are the
> problems I found ...
After those you have the same problem in http_read_headers(), combined
with not honoring "Connection: close" for HTTP/1.1 keep alive requests.
After fixing the first three (all but the header), it did the right
thing.
--
James Antill -- james@and.org
Need an efficient and powerful string library for C?
http://www.and.org/vstr/
| |
| David Schwartz 2005-04-20, 8:58 pm |
|
"Jeff" <joesiege@gmail.com> wrote in message
news:1114004662.770175.245640@l41g2000cwc.googlegroups.com...
> I tried this. Every single close returns -1 with errno set to EBADF
>
> Jeff
Does it do this on the runs where you have the problem? It should do
that on a run that causes no problems.
DS
| |
| Moshe Jacobson 2005-04-21, 3:59 pm |
| David Schwartz <davids@webmaster.com> wrote:
> int i;
> for(i=getdtablesize(); i>2; i++)
> close(i);
> This should close any inherited file descriptors that you don't want.
> Ignore errors from 'close'.
I think you mean i-- as the third argument to for(), right?
Moshe
--
*** SPAM BLOCK: Remove bra before replying! ***
http://runslinux.net :: moshe at runslinux dot net :: AIM: Jehsom
| |
| David Schwartz 2005-04-21, 8:57 pm |
|
"Moshe Jacobson" <bramoshe@runslinux.net> wrote in message
news:4265a328@mustang.speedfactory.net...
> David Schwartz <davids@webmaster.com> wrote:
>
>
> I think you mean i-- as the third argument to for(), right?
>
> Moshe
Yes.
DS
|
|
|
|
|