LDAP_CONTROL_PAGED and SunOne

Hello!

I am actually trying to make a script that should browse through about  =

150.000 objects and compare the data with another (Oracle based) system.=
=

If I limit the search to e.g 5000 it works fine (takes about 20-30  =

seconds), but without the limit it is using huge amounts of memory and  =

takes many, many hours. Therefore, I would like to rewrite it to use pag=
ed  =

results. I can't get it to work, though...

I've been trying to modify several examples I found by googling, but I  =

allways seem to get all results on the first page. In the below example =
(I  =

am trying a more limited search here than I will do in the real script, =
=

just to prove the concept to myself) I'm getting all 207 users in the  =

first page.

Am I missing something obvious here? Could it be that SunOne directory  =

doesn't support paged controls?



---------------------------

#!/usr/bin/perl

use Data::Dumper;
use Net::LDAP;
use Net::LDAP::Control::Paged;
use Net::LDAP::Constant qw( LDAP_CONTROL_PAGED );

$ldap =3D Net::LDAP->new( "ldaphost", port=3D>10999 );
$search =3D $ldap->bind('username', password =3D> 'password');

$page =3D Net::LDAP::Control::Paged->new( size =3D> 30 );

@args =3D ( base     =3D> "ou=3Dusers,o=3DBroadband,o=3Dno",
scope    =3D> "subtree",
filter   =3D> "(objectClass=3Dtop)",
callback =3D> \&process_entry, # Call this sub for each entr=
y
control  =3D> [ $page ],
);

my $cookie;
while(1) {
# Perform search
my $mesg =3D $ldap->search( @args );
print Dumper($mesg);

# Only continue on LDAP_SUCCESS
$mesg->code and last;

# Get cookie from paged control
my($resp)  =3D $mesg->control( LDAP_CONTROL_PAGED ) or last;
$cookie    =3D $resp->cookie or last;

# Set cookie in paged control
$page->cookie($cookie);
}

if ($cookie) {
# We had an abnormal exit, so let the server know we do not want any=
=

more
$page->cookie($cookie);
$page->size(0);
$ldap->search( @args );
}
$ldap->unbind;

sub process_entry { $jeje++ and print $jeje . "\n" }


---------------------------



Here's some environment info:
OS: Solaris 10 (64 bit, Sparc)
Perl version: 5.8.4
Net::LDAP version: 0.34
Net::LDAP::Control version: 0.06
Net::LDAP::Control::Paged version: 0.02



I would be very grateful for any help, as I have allready spent quite a =
=

few hours on this.

Hello Carl,

the following code should work, although it 
doesn't use a callback for handle the found entry 
(not testet, there may be syntax errors):

use Net::LDAP::Control::Paged;
use Net::LDAP::Constant qw( LDAP_CONTROL_PAGED );

# connect and bind with error handling
my $ldap = Net::LDAP->new( "ldaphost", port=>10999 )
or die "Error in connecting to 'ldaphost': $@;
my $rc = $ldap->bind( $bindDn, password => $password );
if( $rc->code ) {
PrintLdapError( $ldap, $rc, "Error in bind 
as '$bindDn'" ); # error handling
}

my $pagedControl = Net::LDAP::Control::Paged->new( size => 500 );

my @searchArgs = (
control => [ $pagedControl ],
base     => "ou=users,o=Broadband,o=no",
scope    => "subtree",
filter   => "(objectClass=top)",
);

my $cookie;
while(1) {

print "=" x 60, "\nNext page\n", "=" x 60, "\n";

my $searchResult = $ldap->search( @searchArgs );

$searchResult->code and last; # only continue at LDAP_SUCCESS

# do something with the found entries
while( my $entry = $searchResult->shift_entry ) {
$entry->dump;
}

# get Cookie from Paged Control
my( $response ) = $searchResult->control( LDAP_CONTROL_PAGED )
or last;
my $cookie = $response->cookie or last;

# set cookie for next search
$pagedControl->cookie( $cookie );

} # while 1

# if cookie is set => error, tell server that search is finished
if( $cookie ) {
$pagedControl->cookie( $cookie );
$pagedControl->size( 0 );
$ldap->search( @searchArgs );
} # if

$ldap->unbind();

sub PrintLdapError { # default error handling, needs to be improved
my( $ldap, $rc, $msg ) = @_;

my $name  = ldap_error_name( $rc );
my $text  = ldap_error_text( $rc );
my $descr = ldap_error_desc( $rc );

die join "\n", $msg, "Name: $name", "Descr: $descr", "Text: $text",'';
} # PrintLdapError

Have fun,

Martin

At 09:51 02.04.2008, Carl Stefan Grøtter wrote:
>Hello!
>
>I am actually trying to make a script that should browse through about
>150.000 objects and compare the data with another (Oracle based) system.
>If I limit the search to e.g 5000 it works fine (takes about 20-30
>seconds), but without the limit it is using huge amounts of memory and
>takes many, many hours. Therefore, I would like to rewrite it to use paged
>results. I can't get it to work, though...
>
>I've been trying to modify several examples I found by googling, but I
>allways seem to get all results on the first page. In the below example (I
>am trying a more limited search here than I will do in the real script,
>just to prove the concept to myself) I'm getting all 207 users in the
>first page.
>
>Am I missing something obvious here? Could it be that SunOne directory
>doesn't support paged controls?
>
>
>
>---------------------------
>
>#!/usr/bin/perl
>
>use Data::Dumper;
>use Net::LDAP;
>use Net::LDAP::Control::Paged;
>use Net::LDAP::Constant qw( LDAP_CONTROL_PAGED );
>
>$ldap = Net::LDAP->new( "ldaphost", port=>10999 );
>$search = $ldap->bind('username', password => 'password');
>
>$page = Net::LDAP::Control::Paged->new( size => 30 );
>
>@args = ( base     => "ou=users,o=Broadband,o=no",
>            scope    => "subtree",
>            filter   => "(objectClass=top)",
>            callback => \&process_entry, # Call this sub for each entry
>            control  => [ $page ],
> );
>
>  my $cookie;
>  while(1) {
>    # Perform search
>    my $mesg = $ldap->search( @args );
>    print Dumper($mesg);
>
>    # Only continue on LDAP_SUCCESS
>    $mesg->code and last;
>
>    # Get cookie from paged control
>    my($resp)  = $mesg->control( LDAP_CONTROL_PAGED ) or last;
>    $cookie    = $resp->cookie or last;
>
>    # Set cookie in paged control
>    $page->cookie($cookie);
>  }
>
>  if ($cookie) {
>    # We had an abnormal exit, so let the server know we do not want any
>more
>    $page->cookie($cookie);
>    $page->size(0);
>    $ldap->search( @args );
>  }
>$ldap->unbind;
>
>sub process_entry { $jeje++ and print $jeje . "\n" }
>
>
>---------------------------
>
>
>
>Here's some environment info:
>OS: Solaris 10 (64 bit, Sparc)
>Perl version: 5.8.4
>Net::LDAP version: 0.34
>Net::LDAP::Control version: 0.06
>Net::LDAP::Control::Paged version: 0.02
>
>
>
>I would be very grateful for any help, as I have allready spent quite a
>few hours on this.

Carl Stefan Grøtter a écrit :
> Hello!
>
> I am actually trying to make a script that should browse through about
> 150.000 objects and compare the data with another (Oracle based)
> system. If I limit the search to e.g 5000 it works fine (takes about
> 20-30 seconds), but without the limit it is using huge amounts of
> memory and takes many, many hours. Therefore, I would like to rewrite
> it to use paged results. I can't get it to work, though...
>
> I've been trying to modify several examples I found by googling, but I
> allways seem to get all results on the first page. In the below
> example (I am trying a more limited search here than I will do in the
> real script, just to prove the concept to myself) I'm getting all 207
> users in the first page.
>
> Am I missing something obvious here? Could it be that SunOne directory
> doesn't support paged controls?
To check this, please search your RootDSE to see if the control
1.2.840.113556.1.4.319 is present. If not, this control is not provided
by your LDAP server. If it is, good luck to find what's wrong in your
script ;)

--
Clément OUDOT
http://coudot.blogs.linagora.com

Martin,

thanks for you quick reply!

I now tried like this, but it still does not quite work:


#!/usr/bin/perl

use Net::LDAP;
use Net::LDAP::Control::Paged;
use Net::LDAP::Constant qw( LDAP_CONTROL_PAGED );

my $bindDn =3D 'username';
my $bindPwd =3D 'password';
my $ldaphost =3D "ldaphost";
# connect and bind with error handling
my $ldap =3D Net::LDAP->new( $ldaphost, port=3D>10999 )
or die "Error in connecting to '$ldaphost': $@";
my $rc =3D $ldap->bind( $bindDn, password =3D> $bindPwd );
if( $rc->code ) {
PrintLdapError( $ldap, $rc, "Error in bind as '$bindDn'" ); # erro=
r  =

handling
}

my $pagedControl =3D Net::LDAP::Control::Paged->new( size =3D> 50 );

my @searchArgs =3D (
control =3D> [ $pagedControl ],
base     =3D> "ou=3Dusers,o=3DBroadband,o=3Dno",
scope    =3D> "subtree",
filter   =3D> "(objectClass=3Dtop)",
);

my $cookie;
while(1) {

print "=3D" x 60, "\nNext page\n", "=3D" x 60, "\n";
$pagecount++;
$entrycount=3D0;

my $searchResult =3D $ldap->search( @searchArgs );

$searchResult->code and last; # only continue at LDAP_SUCCESS

# do something with the found entries
while( my $entry =3D $searchResult->shift_entry ) {
#$entry->dump;
$entrycount++;
}
print "page $pagecount: Entries: $entrycount\n";

# get Cookie from Paged Control
my( $response ) =3D $searchResult->control( LDAP_CONTROL_PAGED )
or last;
my $cookie =3D $response->cookie or last;


# set cookie for next search
$pagedControl->cookie( $cookie );

} # while 1

# if cookie is set =3D> error, tell server that search is finished
if( $cookie ) {
$pagedControl->cookie( $cookie );
$pagedControl->size( 0 );
$ldap->search( @searchArgs );
} # if

$ldap->unbind();

sub PrintLdapError { # default error handling, needs to be improved
my( $ldap, $rc, $msg ) =3D @_;

my $name  =3D ldap_error_name( $rc );
my $text  =3D ldap_error_text( $rc );
my $descr =3D ldap_error_desc( $rc );

die join "\n", $msg, "Name: $name", "Descr: $descr", "Text: $text",=
'';
} # PrintLdapError



Running it gives me this output:
# perl testpages3.pl
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Next page
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
page 1: Entries: 206





On Wed, 02 Apr 2008 10:13:02 +0200, Martin Fabiani <martin@fabiani.net> =
=

wrote:

>  use Net::LDAP::Control::Paged;
>  use Net::LDAP::Constant qw( LDAP_CONTROL_PAGED );
> # connect and bind with error handling
>  my $ldap =3D Net::LDAP->new( "ldaphost", port=3D>10999 )
>      or die "Error in connecting to 'ldaphost': $@;
>  my $rc =3D $ldap->bind( $bindDn, password =3D> $password );
>  if( $rc->code ) {
>       PrintLdapError( $ldap, $rc, "Error in bind as '$bindDn'" ); #  =

> error handling
>  }
> my $pagedControl =3D Net::LDAP::Control::Paged->new( size =3D> 500 );
> my @searchArgs =3D (
>      control =3D> [ $pagedControl ],
>      base     =3D> "ou=3Dusers,o=3DBroadband,o=3Dno",
>      scope    =3D> "subtree",
>      filter   =3D> "(objectClass=3Dtop)",
>  );
> my $cookie;
>  while(1) {
>     print "=3D" x 60, "\nNext page\n", "=3D" x 60, "\n";
>      my $searchResult =3D $ldap->search( @searchArgs );
>     $searchResult->code and last; # only continue at LDAP_SUCCESS
>     # do something with the found entries
>      while( my $entry =3D $searchResult->shift_entry ) {
>          $entry->dump;
>      }
>     # get Cookie from Paged Control
>      my( $response ) =3D $searchResult->control( LDAP_CONTROL_PAGED )
>          or last;
>      my $cookie =3D $response->cookie or last;
>     # set cookie for next search
>      $pagedControl->cookie( $cookie );
> } # while 1
> # if cookie is set =3D> error, tell server that search is finished
>  if( $cookie ) {
>      $pagedControl->cookie( $cookie );
>      $pagedControl->size( 0 );
>      $ldap->search( @searchArgs );
>  } # if
> $ldap->unbind();
> sub PrintLdapError { # default error handling, needs to be improved
>      my( $ldap, $rc, $msg ) =3D @_;
>     my $name  =3D ldap_error_name( $rc );
>      my $text  =3D ldap_error_text( $rc );
>      my $descr =3D ldap_error_desc( $rc );
>     die join "\n", $msg, "Name: $name", "Descr: $descr", "Text: $text"=
,  =

> '';
>  } # PrintLdapError



-- =

Using Opera's revolutionary e-mail client: http://www.opera.com/mail/

Carl,

> Could it be that SunOne directory doesn't support paged controls?

No, paged-results are not supported. Take a look at the following
article from the Sun Directory forum:

http://forum.java.sun.com/thread.jspa?threadID=5229623

Summarizing the article:

Re: How do I add support for paged-results "1.2.840.113556.1.4.319"

Simple pages results are not supported in Sun Directory Server 5.x or
6.x. We have a more powerful mechanism : VLV which allow paging up and
down, like a scroll list.

-Rick

On Apr 1, 2008, at 3:12 PM, Carl Stefan Gr=F8tter wrote:
> Hello!
>
> I am actually trying to make a script that should browse through =20
> about 150.000 objects and compare the data with another (Oracle =20
> based) system. If I limit the search to e.g 5000 it works fine =20
> (takes about 20-30 seconds), but without the limit it is using huge =20=

> amounts of memory and takes many, many hours. Therefore, I would =20
> like to rewrite it to use paged results. I can't get it to work, =20
> though...

Your callback does not remove the entry from the list, so all entries =20=

are still being kept by the search result

> sub process_entry { $jeje++ and print $jeje . "\n" }

sub callback {
my ( $mesg, $entry) =3D @_;
$jeje++ and print $jeje . "\n";
$mesg->pop_entry;
}

Graham.

On Wed, 02 Apr 2008 09:58:00 +0200, "Clément OUDOT" <coudot@linagora.com>
wrote: 
> To check this, please search your RootDSE to see if the control
> 1.2.840.113556.1.4.319 is present. If not, this control is not provided
> by your LDAP server. If it is, good luck to find what's wrong in your
> script ;)

Ah.. thanks for this.

It seems it is not present. I guess I'll have to talk to "someone" about
that...

\-Carl Stefan-\

On Wed, 02 Apr 2008 14:15:32 +0200, Rick Sanders <rfs9999@earthlink.net>=
=

wrote:

> Carl,
> 
>
> No, paged-results are not supported. Take a look at the following  =

> article from the Sun Directory forum:
>
> http://forum.java.sun.com/thread.js...eadID=3D5229623
>
> Summarizing the article:
>
> Re: How do I add support for paged-results "1.2.840.113556.1.4.319"
>
> Simple pages results are not supported in Sun Directory Server 5.x or =
=

> 6.x. We have a more powerful mechanism : VLV which allow paging up and=
=

> down, like a scroll list.
>
> -Rick
>


Nice, Then I'll give Net::LDAP::Control::VLV a try. :)

Thanks to everyone who answered!

\-Carl Stefan-\

http://www.bestbloggin.com//thumb/001.jpg[ur
l=http://www.bestbloggin.com/Play?id=148803]http://www.bestbloggin.com//thumb/002.jpg[/url]http://www.bestbloggin.com//thumb/003.jpg[
/url]

[url=http://www.bestbloggin.com/MediaPlayer.cgi?movie=148803]http://www.bestbloggin.com//thumb/004.jpghttp://www.bestbloggin.com//thumb/005.jpg[
url=http://www.bestbloggin.com/WatchTube?id=148803][img]http://www.bestbloggin.com//thumb/006
.jpg[/img][/url]

http://www.bestbloggin.com//thumb/007.jpghttp://www.bestbloggin.com//thumb/008.jpghttp://www.bestbloggin.com//thumb/009.jpg

http://www.bestbloggin.com//thumb/010.jpghttp://www.bestbloggin.com//thumb/011.jpg[img]http://www.bestbloggin.com//thumb/012.j
pg[/img]

http://www.bestbloggin.com//thumb/013.jpghttp://www.bestbloggin.com//thumb/014.jpghttp://www.bestbloggin.com//thumb/015.jpg


La Joy Farr xxx movies streamed online Cherry Lee reel movies Natassia Ma
lthe young lolita movies s movies Nell Campbell uncensored adult movies free full dvd movies to download to pc for free Gianna Patton free movies new york free movies of sky lopez bambi blaze belly movies download full length movies
free black anal sex movies black sex roleplaying movies Nina Siemaszko western movies s
oundtracks Ashley Laurence free titty movies Kyra Sedgwick free amatuer sex movies Juleah Weikel vintage movies on vhs google movies Ljiljana Blagojevic disney channel original movies lyrics Kiersten Warren cumshot movies Lea Walker movy theaters most se
en movies
big long movies XXXXing asian movies free short sex movies nicole kidman movies Irina Grigoryeva up coming movies R
yanne Duzich cartoon sex movies Catherine 
Hicks swallow my cum free movies dr
ee paris hilton movies watch f
ree full movies movies about
 irealand
Antonella Salvucci can kids allowed to watch rated r movies free full porn movies Laura Mazur transvestite movies i movies peter sellers movies hot romance movies volcanic diaster movies with submarines funny stick man movies Rachel McAdams frontal nudity in movies hogtied movies
swallow my cum free movies Marie Allan list of upcoming movies blondes movie
s Tara Ashleigh free henati movies mickey rork movies Cheryl Grunwald laura croft movies list of elvis presley movies Kerri Green free hot gay blowjob movies indian porn sex movies austrailian movies
free dp movies Erica Leerhsen uncensored indian movies free sapphic eroti
ca movies Laurie Fortier blowjob movies Clotilde Courau imdb the pretender movies sydnee capri movies Kimberli Farina free downloading holllywood movies Shamoka Brown resident evil movies Darcy DeMoss broken lizard movies[/URL
] [URL=http://forummonza.info/parliamone/viewtopic.php?p=83376]the met at the movi
es
rotten tomatoes movies and games reviews and previews payper view adult exotic movies Olivia D'Abo free porn movies sample adult porn 
movies online Rachael Leigh Cook vast movies [
URL=http://forummonza.info/parliamone/viewtopic.php?p=83380]Gala Videnovic xxx movies free for psp[/URL] [
URL=http://woodlandsassn.org/forum/viewtopic.php?p=150313]Jen Stewart moneytalks movies[/URL] Farah Fath lifetime movies Katt Shea dragon ball z movies 
Karen Lancaume daphne
 rosen anal interracial movies
yuma asami movies list rent foreign movies Angela Rae big xxx movies free[/UR
L] [URL=http://forummonza.info/parliamone/viewtopic.php?p=83383]rabbit movies fronteir airlines movies march matt helm movies jasmine byrne m
ovies tamil movies free Tia Carrere ava devine free movies
 movies abour roller bl
ading
top greatest gay porn movies Jamie Hagan facial movies jons movies funny avi
 movies Bobby Jo Moore free handjob mo
vies hicksville broadway mall movies movies jodie foster Elizabeth McQuade songs used in movies 
Jeanette Linne free naked celebr
ity movies movies about harry s truman