PERL Beginners - What is the best way to release memory from data structure?

Author

What is the best way to release memory from data structure?

Chung Ley

2005-04-19, 3:56 pm

Hi,

I have a complex data structure using hash of hash of hash that I need =
to hold one unit of data that I need to process. After finishing =
processing this unit, I need to go to the 2nd unit; each unit will be =
approx. 10Meg of data.

What is the "quick" and "efficient" way for me to release the memory of =
the previous unit so that I can go and reuse it again?

I tried a couple of following methods and neither seem to work.... =
Please let me know if I am doing something wrong or should I take =
another approach...

1. Just set the hash to "empty"
my %datahash;
while (moredetail) {
$datahash{$level1}{$level2}{$level3}{$le
vel4} =3D $something;
}
&processdata(\%datahash);

%datahash =3D ();

2. Putting the function in a separate file and invoking it from the main =
program.

Thanks...

--Chung

Offer Kaye

2005-04-19, 8:55 pm

On 4/19/05, Ley, Chung wrote:
> Hi,
>=20
> I have a complex data structure using hash of hash of hash that I need to=
hold one unit of=20
> data that I need to process. After finishing processing this unit, I nee=
d to go to the 2nd=20
> unit; each unit will be approx. 10Meg of data.
>=20
> What is the "quick" and "efficient" way for me to release the memory of t=
he previous unit=20
> so that I can go and reuse it again?
>=20
> I tried a couple of following methods and neither seem to work.... Pleas=
e let me know if I=20
> am doing something wrong or should I take another approach...
>=20
> 1. Just set the hash to "empty"
> my %datahash;
> while (moredetail) {
> $datahash{$level1}{$level2}{$level3}{$le
vel4} =3D $something;
> }
> &processdata(\%datahash);
>=20
> %datahash =3D ();
>=20

This doesn't release the memory? How do you know?
Have you tried using Devel::Size [1] to see the size of %datahash
after the release?
What about Devel::Monitor [2]?
What happens if you do as the release:
$datahash{$level1}{$level2}{$level3}{$le
vel4} =3D "";
Does that help?

If all else fails, you might consider using tied hashes. Read:
http://perldoc.perl.org/functions/tie.html
and
http://perldoc.perl.org/Tie/Hash.html

[1] http://search.cpan.org/dist/Devel-Size/
[2] http://search.cpan.org/dist/Devel-Monitor/

HTH,
--=20
Offer Kaye

Zentara

2005-04-20, 3:56 pm

On Tue, 19 Apr 2005 10:32:04 -0700, chung.ley@amd.com (Chung Ley) wrote:

>Hi,
>
>I have a complex data structure using hash of hash of hash that I need to hold one unit of data that I need to process. After finishing processing this unit, I need to go to the 2nd unit; each unit will be approx. 10Meg of data.
>
>What is the "quick" and "efficient" way for me to release the memory of the previous unit so that I can go and reuse it again?
>
>I tried a couple of following methods and neither seem to work.... Please let me know if I am doing something wrong or should I take another approach...
>
>1. Just set the hash to "empty"
>my %datahash;
>while (moredetail) {
> $datahash{$level1}{$level2}{$level3}{$l
evel4} = $something;
>}
>&processdata(\%datahash);
>
>%datahash = ();
>
>
>2. Putting the function in a separate file and invoking it from the main program.
>
>Thanks...

I've found it is quite hard to consistently predict when
Perl will release memory back to the system, it generally will
keep it in it's own "memory pool" for reuse.

However, it can be done.
The basics are
1. undef it.
2. make sure it goes "out of scope"
3. if it's an object, make sure it's "ref count" goes to 0.

Read this:
http://www.perlmonks.org?node_id=336883

And here is one of the few examples I've been shown,
which actually does return memory back to the system.

On my setup, going "out of scope" dosn't seem to matter
in this case. So run "top" , and then run this script, and
watch how it's memory usage works.
The interplay between "undef" and "going out of scope"
is important.

This example is very simple, it may not work as easily
in big complex scripts.

If you want to learn more about this topic, do a google
search for "memory garbage collection".

#!/usr/bin/perl -w
use strict;
$| = 1;
{ #start of scope
my $string;
for ( 1 .. 100000 ) {
$string .= ( 'x' x 1000 );
}
print "press enter to release";
<>;
undef $string; # note that memory does not get released

print "undefined but still in scope of sub, hit enter\n";
<>;

# if the variable only goes out of scope.
# you *need* to undef it!
} #end of scope

print "ok,out of scope, press enter to exit";
<>;

__END__

--
I'm not really a human, but I play one on earth.
http://zentara.net/japh.html

Sponsored Links