PERL Beginners - Memory Usage of my Script

Author

Memory Usage of my Script

Yitzle

2008-01-03, 4:01 am

Hi.
I've got two scripts I am running and they both consume large amounts
of memory (10MB).
How would I go about finding where the memory is being used and
figuring out how to reduce the memory footprint.

Both scripts start off with:

#!/usr/bin/perl
use warnings;
use strict;
use WWW::Mechanize;
use HTTP::Cookies;
use Term::ReadKey;

Would removing a "use" help? I can disable strict and warnings while
not editing the script. And Term::ReadKey isn't really needed.
Is it possible that Mechanize is using a really large amount of RAM? I
set the stack_depth to 1.

Spiros Denaxas

2008-01-03, 8:01 am

On Jan 3, 4:10 am, yit...@users.sourceforge.net (Yitzle) wrote:
> Hi.
> I've got two scripts I am running and they both consume large amounts
> of memory (10MB).
> How would I go about finding where the memory is being used and
> figuring out how to reduce the memory footprint.
>
> Both scripts start off with:
>
> #!/usr/bin/perl
> use warnings;
> use strict;
> use WWW::Mechanize;
> use HTTP::Cookies;
> use Term::ReadKey;
>
> Would removing a "use" help? I can disable strict and warnings while
> not editing the script. And Term::ReadKey isn't really needed.
> Is it possible that Mechanize is using a really large amount of RAM? I
> set the stack_depth to 1.

Hello,

You can use Devel::DumpSizes [1] in order to trace the memory usage of
each variable at a given point during your script.
Also, you can check out for cyclical memory objects which consume
larger amounts of memory using Devel::Cycle [2].
This should give you some useful insight on structures you can limit.

Hope this helps,

Spiros

[1] http://search.cpan.org/perldoc?Devel::DumpSizes
[2] http://search.cpan.org/perldoc?Devel::Cycle

Jenda Krynicky

2008-01-03, 8:01 am

From: yitzle <yitzle@users.sourceforge.net>
> Hi.
> I've got two scripts I am running and they both consume large amounts
> of memory (10MB).
> How would I go about finding where the memory is being used and
> figuring out how to reduce the memory footprint.
>
> Both scripts start off with:
>
> #!/usr/bin/perl
> use warnings;
> use strict;
> use WWW::Mechanize;
> use HTTP::Cookies;
> use Term::ReadKey;
>
> Would removing a "use" help? I can disable strict and warnings while
> not editing the script. And Term::ReadKey isn't really needed.

use strict and use warnings will not make a big difference and
removing them is definitely not worth it.

> Is it possible that Mechanize is using a really large amount of RAM? I
> set the stack_depth to 1.

Of course it is. Especially if the page(s) are big. The module has to
parse the HTML and the resulting structure may be quite big.

And really ... I would not call 10MB a large amount of memory
nowadays.

Jenda
===== Jenda@Krynicky.cz === http://Jenda.Krynicky.cz =====
When it comes to wine, women and song, wizards are allowed
to get drunk and croon as much as they like.
-- Terry Pratchett in Sourcery

Yitzle

2008-01-17, 7:02 pm

Hi. Sorry for resurrecting such an old conversation.
I don't have any Objects, so I don't think I can have any cycles.
DumpSizes tells me I'm only using around 200KB RAM.
Yet top reports close to 10MB, mostly in the DATA segment.
Where is that memory going? Is it probably the includes? I don't see
where else it can be.

On 1/3/08, Spiros Denaxas <s.denaxas@gmail.com> wrote:
> On Jan 3, 4:10 am, yit...@users.sourceforge.net (Yitzle) wrote:
>
> Hello,
>
> You can use Devel::DumpSizes [1] in order to trace the memory usage of
> each variable at a given point during your script.
> Also, you can check out for cyclical memory objects which consume
> larger amounts of memory using Devel::Cycle [2].
> This should give you some useful insight on structures you can limit.
>
> Hope this helps,
>
> Spiros
>
> [1] http://search.cpan.org/perldoc?Devel::DumpSizes
> [2] http://search.cpan.org/perldoc?Devel::Cycle

Tom Phoenix

2008-01-17, 7:02 pm

On Jan 17, 2008 3:36 PM, yitzle <yitzle@users.sourceforge.net> wrote:

> DumpSizes tells me I'm only using around 200KB RAM.
> Yet top reports close to 10MB, mostly in the DATA segment.
> Where is that memory going? Is it probably the includes? I don't see
> where else it can be.

As it runs, your perl binary allocates and deallocates memory. In
general, when it needs more, it asks the OS for more; when it no
longer needs some, it doesn't give it back.

As a result, you may have more memory allocated than your program
needs now, because it needed it a moment ago. And, of course, it may
need it again in another moment; that's part of why it doesn't give
the memory back.

This may sound inefficient of memory, but it's probably not. That's
partially because most modern operating systems use virtual memory,
instead of real memory; so, mostly, you only need as much real memory
as your programs are actually using at any given moment.

If you have many similar programs running, especially if they each run
for many minutes at a time, there probably is some way that they can
be reorganized to use less memory as a group. Let us know if that's
the case.

If your system is so low on memory that a single program requiring
10MB causes shortages, you should probably get more memory. (Ah, for
the days when 16k was a memory upgrade!)

If your system isn't running low on memory, you probably shouldn't
worry about it.

If, despite everyone's advice, you spend the next couple of months
tuning your program to use less RAM, you'll probably find that you've
made it much slower and less efficient as well. (It was using some of
that memory to hold values that it would otherwise need to
recalculate, you see.) It would be more efficient in the long run to
spend your time working at the car wash and earning enough in tips to
buy that extra memory, which you can then use to play NetHack with the
rest of us.

Now you see why Perl programmers don't spend much time worrying about
memory management.

Cheers!

--Tom Phoenix
Stonehenge Perl Training

Sponsored Links