waiting for search results

What's the recommended method for testing to see if a search has produced
any results?

I've got a strange problem and I'm pretty certain that there's some kind
of synchronization issue between my search and reads.

In more detail and for background information, I'm working on an
LDAP-based blacklist plugin for SpamAssassin. By default, the plugin runs
in a non-persistent mode, such that new LDAP sessions are established for
each incoming message, queries are run, and the session is destroyed. This
all seems to work pretty well, but its a lot of sessions to deal with.

For efficiency I am trying to get the processes to run in persistent mode,
so that a daemonized instance of SA can reuse a single LDAP session for
the lifetime of that process (each instance of SA gets its own session).
This seems to work for a little while, but eventually it develops a
timeout condition where the LDAP search results don't get caught, so the
plugin hangs which causes a cascade of timeout-related problems. My
current (development) code has:

#
# issue the LDAP search
#
$permsgstatus->{ldap_search_result} = $self->{ldap_session}->search(
base => $self->{ldap_search_base},
filter => $permsgstatus->{ldap_search_filter},
scope => $self->{ldap_search_scope},
deref => $self->{ldap_search_deref},
timelimit => $self->{ldap_timeout},
attrs => [$permsgstatus->{ldap_resource_attr}]);

#
# see if there was a problem
#
if ((! defined $permsgstatus->{ldap_search_result}) ||
($permsgstatus->{ldap_search_result}->error ne "Success")) {

#
# report the error if it's defined
#
if (defined $permsgstatus->{ldap_search_result}) {

dbg ("ldapBlacklist\: search failed with \"" .
$permsgstatus->{ldap_search_result}->error . "\"");
}

#
# disconnect the session politely
#
dbg ("ldapBlacklist\: unable to proceed " .
"... terminating");

$self->destroy_ldap_session($permsgstatus);

#
# kill the module so that nothing more happens
#
die;
}

I'm not getting any hits on this when persistency is enabled and when a
failure occurs. I checked the LDAP logs and the queries are definitely
getting processed, so my guess here is that I'm running into some kind of
synchronization problem and I need to wait for the query to be answered or
errored as appropriate.

Any suggestions here?

Thanks

Here's a demonstration of the problem. First a successful reuse:

May 17 17:57:40 goose spampd[29529]: debug: ldapBlacklist: input data
found ... proceeding with searches
May 17 17:57:40 goose spampd[29529]: debug: ldapBlacklist:   searching for
(objectclass=*)
May 17 17:57:41 goose spampd[29529]: debug: ldapBlacklist: LDAP probe
succeeded for
ldap://localhost:389/ou=Mail-Filters,ou=Services,dc=ehsco,dc=com/


The next message that came into the same process crapped out:

May 17 17:58:14 goose spampd[29529]: debug: ldapBlacklist: input data
found ... proceeding with searches
May 17 17:58:14 goose spampd[29529]: debug: ldapBlacklist:   searching for
(objectclass=*)
May 17 18:02:59 goose spampd[29529]: Failed to run LDAP_RDNS_FROM_WHITE
SpamAssassin test, skipping:    (Timed out! )

Note the ~4 minute delay between query and timeout.


The next message milked by that process also worked fine, and the existing
ldap session was reused just fine:

May 17 18:09:05 goose spampd[29529]: debug: ldapBlacklist: input data
found ... proceeding with searches
May 17 18:09:05 goose spampd[29529]: debug: ldapBlacklist:   searching for
(objectclass=*)
May 17 18:09:06 goose spampd[29529]: debug: ldapBlacklist: LDAP probe
succeeded for
ldap://localhost:389/ou=Mail-Filters,ou=Services,dc=ehsco,dc=com/


Looking at the ldap logfile however shows that the "failing" query was
actually processed successfully:

May 17 17:58:14 goose slapd[16685]: conn=3857 op=468 SRCH
base="ou=Mail-Filters,ou=Services,dc=ehsco,dc=com" scope=0
filter="(objectClass=*)"
May 17 17:58:14 goose slapd[16685]: conn=3857 op=468 SRCH attr=objectClass
May 17 17:58:14 goose slapd[16685]: conn=3857 op=468 SEARCH RESULT tag=101
err=0 nentries=1 text=

For some reason, Net::LDAP either didn't get the answer, didn't pass the
answer back to my script, or didn't return an error that I could trap.
Even more confusing, this only happens when I make the session persistent,
and it never happens when I create/destroy per-message sessions (but that
doesn't seem to matter, since as demonstrated subsequent messages seem to
still get processed just fine).

If anybody has any ideas here I'd appreciate hearing them.

Thanks

Original message below for context:


Eric A. Hall wrote:

> What's the recommended method for testing to see if a search has
> produced any results?
>
> I've got a strange problem and I'm pretty certain that there's some
> kind of synchronization issue between my search and reads.
>
> In more detail and for background information, I'm working on an
> LDAP-based blacklist plugin for SpamAssassin. By default, the plugin
> runs in a non-persistent mode, such that new LDAP sessions are
> established for each incoming message, queries are run, and the session
> is destroyed. This all seems to work pretty well, but its a lot of
> sessions to deal with.
>
> For efficiency I am trying to get the processes to run in persistent
> mode, so that a daemonized instance of SA can reuse a single LDAP
> session for the lifetime of that process (each instance of SA gets its
> own session). This seems to work for a little while, but eventually it
> develops a timeout condition where the LDAP search results don't get
> caught, so the plugin hangs which causes a cascade of timeout-related
> problems. My current (development) code has:
>
> # # issue the LDAP search # $permsgstatus->{ldap_search_result} =
> $self->{ldap_session}->search( base => $self->{ldap_search_base},
> filter => $permsgstatus->{ldap_search_filter}, scope =>
> $self->{ldap_search_scope}, deref => $self->{ldap_search_deref},
> timelimit => $self->{ldap_timeout}, attrs =>
> [$permsgstatus->{ldap_resource_attr}]);
>
> # # see if there was a problem # if ((! defined
> $permsgstatus->{ldap_search_result}) ||
> ($permsgstatus->{ldap_search_result}->error ne "Success")) {
>
> # # report the error if it's defined # if (defined
> $permsgstatus->{ldap_search_result}) {
>
> dbg ("ldapBlacklist\: search failed with \"" .
> $permsgstatus->{ldap_search_result}->error . "\""); }
>
> # # disconnect the session politely # dbg ("ldapBlacklist\: unable to
> proceed " . "... terminating");
>
> $self->destroy_ldap_session($permsgstatus);
>
> # # kill the module so that nothing more happens # die; }
>
> I'm not getting any hits on this when persistency is enabled and when a
>  failure occurs. I checked the LDAP logs and the queries are definitely
>  getting processed, so my guess here is that I'm running into some kind
>  of synchronization problem and I need to wait for the query to be
> answered or errored as appropriate.
>
> Any suggestions here?
>
> Thanks

On 17/5/05 10:18, Eric A. Hall <ehall@ehsco.com> wrote:

>
> What's the recommended method for testing to see if a search has produced
> any results?
>
> I've got a strange problem and I'm pretty certain that there's some kind
> of synchronization issue between my search and reads.
>
> In more detail and for background information, I'm working on an
> LDAP-based blacklist plugin for SpamAssassin. By default, the plugin runs
> in a non-persistent mode, such that new LDAP sessions are established for
> each incoming message, queries are run, and the session is destroyed. This
> all seems to work pretty well, but its a lot of sessions to deal with.
>
> For efficiency I am trying to get the processes to run in persistent mode,
> so that a daemonized instance of SA can reuse a single LDAP session for
> the lifetime of that process (each instance of SA gets its own session).
> This seems to work for a little while, but eventually it develops a
> timeout condition where the LDAP search results don't get caught, so the
> plugin hangs which causes a cascade of timeout-related problems. My
> current (development) code has:
>
>   #
>   # issue the LDAP search
>   #
>   $permsgstatus->{ldap_search_result} = $self->{ldap_session}->search(
> base => $self->{ldap_search_base},
> filter => $permsgstatus->{ldap_search_filter},
> scope => $self->{ldap_search_scope},
> deref => $self->{ldap_search_deref},
> timelimit => $self->{ldap_timeout},
> attrs => [$permsgstatus->{ldap_resource_attr}]);
>
>   #
>   # see if there was a problem
>   #
>   if ((! defined $permsgstatus->{ldap_search_result}) ||

The search method will generally only return undef if it couldn't send the
operation to the server.

> ($permsgstatus->{ldap_search_result}->error ne "Success")) {

The Message class's error method will block until the search completes. It
looks like it'll return some text returned by the server, which is *not*
useful to test.

Aside: LDAP servers return several bits of info in operation results. One is
the result code (an integer) which describes the precise outcome of the
operation. There's also a text field which can be used by servers for
returning some bit of free text. This text field doesn't contain anything
that should be tested by a program, as the contents of it aren't defined.

So I'd test the value of the code method instead, as this will return a
portable LDAP error number. (0 is OK)

>
> #
> # report the error if it's defined
> #
> if (defined $permsgstatus->{ldap_search_result}) {
>
> dbg ("ldapBlacklist\: search failed with \"" .
> $permsgstatus->{ldap_search_result}->error . "\"");
> }
>
> #
> # disconnect the session politely
> #
> dbg ("ldapBlacklist\: unable to proceed " .
> "... terminating");
>
> $self->destroy_ldap_session($permsgstatus);
>
> #
> # kill the module so that nothing more happens
> #
> die;
>   }
>
> I'm not getting any hits on this when persistency is enabled and when a
> failure occurs. I checked the LDAP logs and the queries are definitely
> getting processed, so my guess here is that I'm running into some kind of
> synchronization problem and I need to wait for the query to be answered or
> errored as appropriate.

Your server could indeed be quite broken, but I'd go and check result codes
properly and see what changes that makes.

Cheers,

Chris

Chris Ridd wrote:

> Your server could indeed be quite broken, but I'd go and check result code
s
> properly and see what changes that makes.

Alright, I changed the "->error ne "Success"" tests to "->code() != 0",
and we'll see how that goes.

I don't remember why I used the error tests in the first place but I
remember doing it for a reason. Perhaps it was that I am getting zero
return codes for some failures. This is OpenLDAP 2-2.1.22-118 on SuSE 9.1
(with patches) and perl 5.8.1 if that makes a difference.

--
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

On 18/5/05 4:28, Eric A. Hall <ehall@ehsco.com> wrote:

> I don't remember why I used the error tests in the first place but I
> remember doing it for a reason. Perhaps it was that I am getting zero
> return codes for some failures. This is OpenLDAP 2-2.1.22-118 on SuSE 9.1
> (with patches) and perl 5.8.1 if that makes a difference.

You have to look at the RFCs (esp RFC 2251 section 4.1.10) to work out what
return codes are "errors" for particular operations. For instance the
compare operation, returns 5 or 6 depending on the assertion and not 0.
Things like timeLimitExceeded indicate a "partial" error...

So it is slightly complicated :-)

Cheers,

Chris

I fixed this (apparently) by adding "eval {local $SIG{ALRM}=sub {die}}"
blocks around the LDAP searches, with an alarm timer equal to the
user-specified LDAP timeout value. If the search doesn't come back on its
own, the code kicks in, and if the hang was fatal it barfs up a return 0
which allows me to exit gracefully. No problems since.


On 5/17/2005 5:18 PM, Eric A. Hall wrote:
> What's the recommended method for testing to see if a search has produced
> any results?
>
> I've got a strange problem and I'm pretty certain that there's some kind
> of synchronization issue between my search and reads.
>
> In more detail and for background information, I'm working on an
> LDAP-based blacklist plugin for SpamAssassin. By default, the plugin runs
> in a non-persistent mode, such that new LDAP sessions are established for
> each incoming message, queries are run, and the session is destroyed. This
> all seems to work pretty well, but its a lot of sessions to deal with.
>
> For efficiency I am trying to get the processes to run in persistent mode,
> so that a daemonized instance of SA can reuse a single LDAP session for
> the lifetime of that process (each instance of SA gets its own session).
> This seems to work for a little while, but eventually it develops a
> timeout condition where the LDAP search results don't get caught, so the
> plugin hangs which causes a cascade of timeout-related problems. My
> current (development) code has:
>
>   #
>   # issue the LDAP search
>   #
>   $permsgstatus->{ldap_search_result} = $self->{ldap_session}->search(
> 	base => $self->{ldap_search_base},
> 	filter => $permsgstatus->{ldap_search_filter},
> 	scope => $self->{ldap_search_scope},
> 	deref => $self->{ldap_search_deref},
> 	timelimit => $self->{ldap_timeout},
> 	attrs => [$permsgstatus->{ldap_resource_attr}]);
>
>   #
>   # see if there was a problem
>   #
>   if ((! defined $permsgstatus->{ldap_search_result}) ||
> 	($permsgstatus->{ldap_search_result}->error ne "Success")) {
>
> 	#
> 	# report the error if it's defined
> 	#
> 	if (defined $permsgstatus->{ldap_search_result}) {
>
> 		dbg ("ldapBlacklist\: search failed with \"" .
> 		 $permsgstatus->{ldap_search_result}->error . "\"");
> 	}
>
> 	#
> 	# disconnect the session politely
> 	#
> 	dbg ("ldapBlacklist\: unable to proceed " .
> 		"... terminating");
>
> 	$self->destroy_ldap_session($permsgstatus);
>
> 	#
> 	# kill the module so that nothing more happens
> 	#
> 	die;
>   }
>
> I'm not getting any hits on this when persistency is enabled and when a
> failure occurs. I checked the LDAP logs and the queries are definitely
> getting processed, so my guess here is that I'm running into some kind of
> synchronization problem and I need to wait for the query to be answered or
> errored as appropriate.
>
> Any suggestions here?
>
> Thanks