PERL POE - graceful shutdown of servers

Author

graceful shutdown of servers

Kevin Scaldeferri

2007-01-02, 10:23 pm

I've been digging through the website and mailing list archives, and
I'm still puzzled about how to achieve a graceful server shutdown.

By this, I mean something similar to 'apachectl graceful', where new
requests are refused, but any existing requests are fully processed
before the server performs shutdown functions and terminates.

My server is an HTTP server using PoCo::Server::TCP, which runs a
fairly simple state machine to gather up all the information required
for each request and then puts it together to construct the reply.
It seems to me like implementing the graceful shutdown requires 3 parts:

1) a signal handler that sets a global "shutting down" flag

2) in my ClientInput callback, to only start the state machine if the
flag is not set

3) somehow determine that all current requests have finished, and
shutdown if the flag is set. This is the part that I'm stuck on.

Any help or suggestions are appreciated.

-kevin

Rocco Caputo

2007-01-03, 4:31 am

On Jan 2, 2007, at 22:25, Kevin Scaldeferri wrote:

> I've been digging through the website and mailing list archives,
> and I'm still puzzled about how to achieve a graceful server shutdown.
>
> By this, I mean something similar to 'apachectl graceful', where
> new requests are refused, but any existing requests are fully
> processed before the server performs shutdown functions and
> terminates.
>
> My server is an HTTP server using PoCo::Server::TCP, which runs a
> fairly simple state machine to gather up all the information
> required for each request and then puts it together to construct
> the reply. It seems to me like implementing the graceful shutdown
> requires 3 parts:
>
> 1) a signal handler that sets a global "shutting down" flag

Good. It should also send a "shutdown" signal to the
POE::Component::Server::TCP instance's alias. This shuts down the
listener but leaves the children running.

> 2) in my ClientInput callback, to only start the state machine if
> the flag is not set

Not necessary. New clients will not be accepted.

> 3) somehow determine that all current requests have finished, and
> shutdown if the flag is set. This is the part that I'm stuck on.

Not necessary. If your server is doing nothing else, it will exit
after the last client disconnects. If you need to trigger a more
complex shutdown after the last client disconnects, then you will
need a shutdown flag and an active connection counter:

a. Send the Server::TCP instance a "shutdown" message.
b. Set the global shutdown flag.
c. If the active session count is 0, commence complex shutdown.
d. Otherwise, increment the active session count from the
ClientConnected callback, and decrement it from the
ClientDisconnected callback.
e. When the active session count reaches 0, and the shutdown flag is
set, begin complex shutdown.

--
Rocco Caputo - rcaputo@pobox.com

Rocco Caputo

2007-01-03, 7:25 pm

The sig() watchers are throwing "signal_shutdown" events to the wrong
session. Your start_server() callback is executed within the context
of the main listener session. PackageStates are used to customize
each connection's session, however, so the signal_shutdown() handler
lives elsewhere.

You can work around this by putting the signal handling in one
session, separate from the TCP server component's sessions.
Something like this untested code:

POE::Session->create(
inline_states => {
_start => sub {
my $kernel = $_[KERNEL];
$kernel->sig( INT => 'signal_shutdown' );
$kernel->sig( TERM => 'signal_shutdown' );
$kernel->sig( HUP => 'signal_shutdown' );
},
signal_shutdown => sub {
my ($kernel, $signal) = @_[KERNEL, ARG0];
debug ("got $signal signal\n");
$kernel->post( server => "shutdown" );
$kernel->sig_handled;
},
},
);

--
Rocco Caputo - rcaputo@pobox.com

On Jan 3, 2007, at 15:27, Kevin Scaldeferri wrote:
[color=darkred]
> Okay, I'm glad to see I was making this more complicated than I had
> to. I'm still a little puzzled, though. Ignoring all the bits
> that actually do stuff, I've got code that looks like this:
>
> POE::Component::Server::TCP->new
> ( Alias => 'server',
> ...
> Started => \&start_server,
> PackageStates => [ main => [..., "signal_shutdown"]],
> );
>
>
> debug ("Starting kernel\n");
> $poe_kernel->run();
> debug ("Kernel stopped\n");
> unlink $pidfile;
> exit;
>
> sub start_server {
> my $kernel = $_[KERNEL];
> $kernel->sig( INT => 'signal_shutdown' );
> $kernel->sig( TERM => 'signal_shutdown' );
> $kernel->sig( HUP => 'signal_shutdown' );
> }
> sub signal_shutdown {
> my ($kernel, $signal) = @_[KERNEL, ARG0];
> debug ("got $signal signal\n");
> $kernel->post( server => "shutdown" );
> $kernel->sig_handled;
> }
>
>
> I can see now that the process is shutting down in a more-or-less
> orderly fashion, because the "Kernel stopped" message appears in my
> logs. Without the signal handlers installed the process died
> before printing that. However, I'm

because the "got
> signal" message doesn't show up, and my pidfile doesn't actually
> get removed from the filesystem.
>
> If I turn on assertions, I see this:
>
> a 'signal_shutdown' event was sent from /home/y/lib/perl5/site_perl/
> 5.8/POE/Loop/PerlSignals.pm at 35 to session 2 (server) but session
> 2 (server) has neither a handler for it nor one for _default
>
> I've tried defining this as both a PackageState and as an
> InlineState, but neither seems to get things to hook up correctly.
>
>
> -kevin
>
>
> On Jan 2, 2007, at 11:02 PM, Rocco Caputo wrote:
>

Kevin Scaldeferri

2007-01-03, 7:25 pm

Okay, I'm glad to see I was making this more complicated than I had
to. I'm still a little puzzled, though. Ignoring all the bits that
actually do stuff, I've got code that looks like this:

POE::Component::Server::TCP->new
( Alias => 'server',
...
Started => \&start_server,
PackageStates => [ main => [..., "signal_shutdown"]],
);

debug ("Starting kernel\n");
$poe_kernel->run();
debug ("Kernel stopped\n");
unlink $pidfile;
exit;

sub start_server {
my $kernel = $_[KERNEL];
$kernel->sig( INT => 'signal_shutdown' );
$kernel->sig( TERM => 'signal_shutdown' );
$kernel->sig( HUP => 'signal_shutdown' );
}
sub signal_shutdown {
my ($kernel, $signal) = @_[KERNEL, ARG0];
debug ("got $signal signal\n");
$kernel->post( server => "shutdown" );
$kernel->sig_handled;
}

I can see now that the process is shutting down in a more-or-less
orderly fashion, because the "Kernel stopped" message appears in my
logs. Without the signal handlers installed the process died before
printing that. However, I'm

because the "got signal"
message doesn't show up, and my pidfile doesn't actually get removed
from the filesystem.

If I turn on assertions, I see this:

a 'signal_shutdown' event was sent from /home/y/lib/perl5/site_perl/
5.8/POE/Loop/PerlSignals.pm at 35 to session 2 (server) but session 2
(server) has neither a handler for it nor one for _default

I've tried defining this as both a PackageState and as an
InlineState, but neither seems to get things to hook up correctly.

-kevin

On Jan 2, 2007, at 11:02 PM, Rocco Caputo wrote:

> On Jan 2, 2007, at 22:25, Kevin Scaldeferri wrote:
>
>
> Good. It should also send a "shutdown" signal to the
> POE::Component::Server::TCP instance's alias. This shuts down the
> listener but leaves the children running.
>
>
> Not necessary. New clients will not be accepted.
>
>
> Not necessary. If your server is doing nothing else, it will exit
> after the last client disconnects. If you need to trigger a more
> complex shutdown after the last client disconnects, then you will
> need a shutdown flag and an active connection counter:
>
> a. Send the Server::TCP instance a "shutdown" message.
> b. Set the global shutdown flag.
> c. If the active session count is 0, commence complex shutdown.
> d. Otherwise, increment the active session count from the
> ClientConnected callback, and decrement it from the
> ClientDisconnected callback.
> e. When the active session count reaches 0, and the shutdown flag
> is set, begin complex shutdown.
>
> --
> Rocco Caputo - rcaputo@pobox.com
>
>

Kevin Scaldeferri

2007-01-03, 7:25 pm

On Jan 3, 2007, at 2:24 PM, Rocco Caputo wrote:

> The sig() watchers are throwing "signal_shutdown" events to the
> wrong session. Your start_server() callback is executed within the
> context of the main listener session. PackageStates are used to
> customize each connection's session, however, so the signal_shutdown
> () handler lives elsewhere.
>
> You can work around this by putting the signal handling in one
> session, separate from the TCP server component's sessions.
> Something like this untested code:

I needed one minor addition

>
> POE::Session->create(
> inline_states => {
> _start => sub {
> my $kernel = $_[KERNEL];

$kernel->set_alias('signal_hander'); # prevent premature
garbage collection

> $kernel->sig( INT => 'signal_shutdown' );
> $kernel->sig( TERM => 'signal_shutdown' );
> $kernel->sig( HUP => 'signal_shutdown' );
> },
> signal_shutdown => sub {
> my ($kernel, $signal) = @_[KERNEL, ARG0];
> debug ("got $signal signal\n");
> $kernel->post( server => "shutdown" );
> $kernel->sig_handled;
> },
> },
> );

With that change, it appears to be working as I wanted.

Thanks!

-kevin

Kevin Scaldeferri

2007-01-03, 10:21 pm

Sponsored Links