PERL Beginners - Working with very large text files

Author

Working with very large text files

ytoledano@gmail.com

2006-01-10, 4:02 am

Hi,
I'm working on a server which writes text to large log files. The
problem is that these files are 1 GB and when I open them with perl,
perl tries to load the entire file into the memory and fails when XP is
out of memory.

I only need to use the text written in the last 5 minutes, which
shouldn't take more then a couple of megs, but since I'm using XP, I
can't use `tail`.

What code doesn't make perl load the entire file into the memory?

Paul Lalli

2006-01-10, 4:02 am

ytoledano@gmail.com wrote:
> Hi,
> I'm working on a server which writes text to large log files. The
> problem is that these files are 1 GB and when I open them with perl,
> perl tries to load the entire file into the memory and fails when XP is
> out of memory.
>
> I only need to use the text written in the last 5 minutes, which
> shouldn't take more then a couple of megs, but since I'm using XP, I
> can't use `tail`.
>
> What code doesn't make perl load the entire file into the memory?

Perl will only load the entire file into memory if you tell it to, by
slurping the entire thing, like:

my @lines = <FILE>;
foreach (@lines) {
process_line($_);
}

If you instead read the file line by line, and discard each one after
it's read, no more than one line will ever be in memory at once:

while (<FILE> ) {
process_line($_);
}

Note that you *must* use while in the above, not foreach. Even though
they look the same, they do *not* operate the same internally.

If you're looking for just the last five lines of the file, you may
also be interested in modules such as Tie::File and File::Tail. (The
former is standard, the latter is available on CPAN)

Paul Lalli

Sponsored Links