PERL Beginners - large Files management

Author

large Files management

Armin Garcia

2007-09-26, 10:00 pm

Hi

well i work with mbox files, but i have a huge mbox file, its 339 MB and
when i processing this file with Mail::MboxParser module, my program breaks
becouse send a message of out of memory....

I think is becouse the module Mail::MboxParser try to up in the cache all
the file but becouse the file is very huge send this out of memory...

My question is, how i can resolve this problem, how i can read a huge file
with the Mail::MboxParser module, and count all the emails that are in the
huge file, is indispensable to use this module....

Here is my code, when i read a mbox file i search the urls

#!/usr/bin/perl

#Modulos
use Mail::MboxParser;
use Net::Nslookup;

#Defaults
$conf="spam.conf";
$spam="Homero";

#inicializando los hash y los arreglos

my $parseropts ={
enable_cache => 1,
enable_grep => 1,
cache_file_name => $spam,
}

open(test,"+> .pruebirilijilla.txt") || die "No pude crear el archivo";
read

my $mb = Mail::MboxParser->new ($spam, decode => 'ALL', parseropts =>
$parseropts);

$contmsj=1;

for my $msg ($mb->get_messages){
push (@subjects,$msg->header->{subject});
push (@froms,$msg->header->{from});
push (@dates,$msg->header->{date});
@correo=$msg->body($msg-find_body);
$msg=join " ",(split " ", $msg);

#Http o https
while($msg =~
/(https?:\/\/\w+[a-zA-Z0-9\.=\/\?\&-_\~:\!\*%@\$]+)/gm){
push(@urls,$1);
#ftp o ftps
}

while($msg=~/(ftps?:\/\/\w+[a-zA-Z0-9\.=\/\?\&-_\~:\!\*%@\$]+/gm){
push(@urls,$1);
}
push(@msg,$msg);
$contmsj++;
}
open(H,"+>Simpsons.txt") || die "No se pudo";
foreach (@urls){
print H $_,"\n";
}

close(H);

thanks and i hope you can help me

Dr.Ruud

2007-09-26, 10:00 pm

"Armin Garcia" schreef:

> well i work with mbox files, but i have a huge mbox file, its 339 MB
> and when i processing this file with Mail::MboxParser module, my
> program breaks becouse send a message of out of memory....

Change-or-convert to maildir-type storage?

--
Affijn, Ruud

"Gewoon is een tijger."

Sponsored Links