| Author |
Sockets: connection refused
|
|
| Oliveira 2006-09-12, 7:03 pm |
| Hi all!
I have a Linux server (FC3) with 3 application that collects several
data from 3 equipments using sockets.
These 3 application are restarted (killed and start) each 15 minutes.
Several hours later, when restarting the application, I get a
'Connection refused' error (in the 3 application).
After this error if we reboot the server everthing goes OK. After
several hours the same error happens.
Does any one has an ideia how to find what is causing this kind of
problema?
I will be grateful for any help.
Regards,
Oliveira
| |
| David Schwartz 2006-09-12, 7:03 pm |
|
Oliveira wrote:
> I have a Linux server (FC3) with 3 application that collects several
> data from 3 equipments using sockets.
> These 3 application are restarted (killed and start) each 15 minutes.
> Several hours later, when restarting the application, I get a
> 'Connection refused' error (in the 3 application).
> After this error if we reboot the server everthing goes OK. After
> several hours the same error happens.
> Does any one has an ideia how to find what is causing this kind of
> problema?
When you get a 'connection refused', it usually means that nothing is
listening on the port you're trying to connect to. There are many
reasons this can happen.
First, check to make sure that the program that listens on that port is
actually running. Then check to see if it has a socket bound to that
port (with 'netstat' or a similar tool). Then check to make sure it
hasn't hung or crashed.
Also, make sure the application will not start up if it fails to bind
to its primary port (the one that you're connecting to). At an absolute
minimum, log the inability to bind to the port on startup and see if
that log entry coincides with your problem.
In other words, troubleshoot the error condition and narrow it down. If
I had to take a total guess, I'd guess that it's a combination of not
setting SO_REUSEPORT and not being diligent in attempting to 'bind' on
startup.
DS
| |
| Maxim Yegorushkin 2006-09-12, 7:03 pm |
| David Schwartz wrote:
> Oliveira wrote:
>
>
> When you get a 'connection refused', it usually means that nothing is
> listening on the port you're trying to connect to. There are many
> reasons this can happen.
>
> First, check to make sure that the program that listens on that port is
> actually running. Then check to see if it has a socket bound to that
> port (with 'netstat' or a similar tool). Then check to make sure it
> hasn't hung or crashed.
In addition to said, it might be a resource leak in the server. On
linux the default limit of open file descriptors may be 1024, the
server might me running out of the them neglecting to close file
descriptors, so that accept fails and RST is sent back to the client,
which causes connect fail with ECONNREFUSED.
| |
| Maxim Yegorushkin 2006-09-13, 4:01 am |
|
Maxim Yegorushkin wrote:
> David Schwartz wrote:
>
> In addition to said, it might be a resource leak in the server. On
> linux the default limit of open file descriptors may be 1024, the
> server might me running out of the them neglecting to close file
> descriptors, so that accept fails and RST is sent back to the client,
> which causes connect fail with ECONNREFUSED.
A quick test to see if the server is leaking fd's would be something
like:
$ lsof -p `pidof server`
| |
| Oliveira 2006-09-13, 7:02 pm |
| First of all thank you very much (David and Maxim) for the answers.
I have one more information that can be useful.
When I do this
netstat | grep http | wc -l
I have more than 1500 lines! apache is running as web server.
This (high) number of http connections can cause the problem related
with socket connection?
| |
| Maxim Yegorushkin 2006-09-13, 7:02 pm |
|
Oliveira wrote:
> First of all thank you very much (David and Maxim) for the answers.
>
> I have one more information that can be useful.
>
> When I do this
> netstat | grep http | wc -l
>
> I have more than 1500 lines! apache is running as web server.
>
> This (high) number of http connections can cause the problem related
> with socket connection?
Do you experience connectivity problems with your apache?
| |
| David Schwartz 2006-09-13, 10:05 pm |
|
Oliveira wrote:
> First of all thank you very much (David and Maxim) for the answers.
>
> I have one more information that can be useful.
>
> When I do this
> netstat | grep http | wc -l
>
> I have more than 1500 lines! apache is running as web server.
>
> This (high) number of http connections can cause the problem related
> with socket connection?
Are these connections to Apache? Are you sure they're to a port Apache
is using?
DS
| |
| Oliveira 2006-09-14, 4:01 am |
|
David Schwartz escreveu:
> Are these connections to Apache? Are you sure they're to a port Apache
> is using?
>
> DS
I think so. The (netstat) lines are similar to the folowing:
tcp 0 0 ::ffff:172.17.11.3:http
::ffff:172.17.11.235:1352 TIME_WAIT
The 172.17.11.3 is the (web) server and the 172.17.11.235 is one client
computer in the plant.
Oliveira
| |
| David Schwartz 2006-09-14, 4:01 am |
|
Oliveira wrote:
> David Schwartz escreveu:
>
>
> I think so. The (netstat) lines are similar to the folowing:
>
> tcp 0 0 ::ffff:172.17.11.3:http
> ::ffff:172.17.11.235:1352 TIME_WAIT
>
> The 172.17.11.3 is the (web) server and the 172.17.11.235 is one client
> computer in the plant.
>
> Oliveira
Is the connection rate supposed to be so high? This indicates a very
high rate of connections, probably 8 per second or so.
DS
| |
| Oliveira 2006-09-14, 7:01 pm |
|
David Schwartz escreveu:
> Is the connection rate supposed to be so high? This indicates a very
> high rate of connections, probably 8 per second or so.
>
> DS
Each cliente computer has a web browser opened (IE) and the web page is
self refreshing each 6 seconds (refreshing is done 10 time per minute).
I change the refresh period from 6 seconds to 90 seconds. Now the
command
netstat | grep http | wc -l
gives somothing about 180. Until now I the "connection refused" problem
never happens, but I still have doubts if the increase of the refresh
period solve the problem.
Once again, thank you for your help.
Oliveira
| |
| David Schwartz 2006-09-14, 7:01 pm |
|
Oliveira wrote:
> Each cliente computer has a web browser opened (IE) and the web page is
> self refreshing each 6 seconds (refreshing is done 10 time per minute).
>
> I change the refresh period from 6 seconds to 90 seconds. Now the
> command
> netstat | grep http | wc -l
>
> gives somothing about 180. Until now I the "connection refused" problem
> never happens, but I still have doubts if the increase of the refresh
> period solve the problem.
Sounds like you were running out of sockets. Most likely, it was the
number of sockets the process was allowed to open. You can either use
fewer sockets (which is what you are doing by lowering the connection
rate) or figure out exactly what resource you are running out of and
increase it.
Is the the Fedora build of apache or did you compile it yourself? Do
you have anything really weird attached to apache as something other
than a CGI? If a CGI, how many instances of it do you typically see in
a 'ps'?
DS
| |
| Oliveira 2006-09-15, 4:01 am |
|
David Schwartz escreveu:
> Sounds like you were running out of sockets. Most likely, it was the
> number of sockets the process was allowed to open. You can either use
> fewer sockets (which is what you are doing by lowering the connection
> rate) or figure out exactly what resource you are running out of and
> increase it.
I think I already did it by increase the refresh period.
>
> Is the the Fedora build of apache or did you compile it yourself? Do
> you have anything really weird attached to apache as something other
> than a CGI? If a CGI, how many instances of it do you typically see in
> a 'ps'?
>
> DS
The apache is build of Fedora (FC3).
The web pages only uses alot of PHP.
The output of
ps -ef | grep apache | wc -l
is 12 processes.
I hope this informations are useful.
Thank you very much.
Oliveira
|
|
|
|