PerlTk - Tk::MListbox leaking memory

Author

Tk::MListbox leaking memory

x3v0-usenet@yahoo.com

2005-05-01, 9:03 pm

This is happening with Tk::MListbox 1.11. I tested it on Fedora Core 2
and Windows 2000.

I believe this problem has been reported before but noone found a
solution.

I am observing memory leak when using MListbox. The leak happens both
when deleting/re-inserting the list and when sorting. I monitored the
memory leak by using "top" in Linux and using the task manager in
Win2k.

The memory seems to go up by the same amount every time. I could not
find the source of this problem. The problem may lie in Tk::Listbox
itself and not MListbox, I can't say for sure.

The below script demonstrates the problem. Everytime the button is
clicked or the column is sorted, the script leaks memory.

########################################
##########

#!/usr/bin/perl -w
use strict;

use Tk;
use Tk::MListbox;

my $mw = Tk::MainWindow->new(-title => 'MListbox Leak');

my $b = $mw->Button(-text => 'Leak', -command =>
\&leak)->pack(-side=>'bottom');
my $mlb = $mw->Scrolled('MListbox', -scrollbars=>'se',
-selectmode=>'extended',)
->pack(-expand=>'true', -fill=>'both', -side=>'top',);

$mlb->columnInsert('end', -text=>'Test');

# setup the initial indices
for(1..10000)
{
$mlb->insert('end',[$_]);
}

MainLoop;

sub leak
{
$mlb->delete(0,'end');

for(0..10000){
$mlb->insert($_,[$_]);
}

$mlb->update;
}

########################################
##########

Rob Seegel

2005-05-01, 9:03 pm

I believe the problem really lies with Listbox, it's just extra
noticable with MListbox, since the same operation is happening in X
times the number of columns you have.

Try the following link, and look at the first four or five threads
that come up in the List. I haven't looked at the C code for Perl Tk in
a while so I can't really say if the patches were somehow "unpatched",
never applied, or if there are additional problems present that the
posted patches didn't cover:

http://groups-beta.google.com/group...coring=d&hl=en&

Rob

x3v0-usenet@yahoo.com wrote:
> This is happening with Tk::MListbox 1.11. I tested it on Fedora Core 2
> and Windows 2000.
>
> I believe this problem has been reported before but noone found a
> solution.
>
> I am observing memory leak when using MListbox. The leak happens both
> when deleting/re-inserting the list and when sorting. I monitored the
> memory leak by using "top" in Linux and using the task manager in
> Win2k.
>
> The memory seems to go up by the same amount every time. I could not
> find the source of this problem. The problem may lie in Tk::Listbox
> itself and not MListbox, I can't say for sure.
>
> The below script demonstrates the problem. Everytime the button is
> clicked or the column is sorted, the script leaks memory.
>
> ########################################
##########
>
> #!/usr/bin/perl -w
> use strict;
>
> use Tk;
> use Tk::MListbox;
>
> my $mw = Tk::MainWindow->new(-title => 'MListbox Leak');
>
> my $b = $mw->Button(-text => 'Leak', -command =>
> \&leak)->pack(-side=>'bottom');
> my $mlb = $mw->Scrolled('MListbox', -scrollbars=>'se',
> -selectmode=>'extended',)
> ->pack(-expand=>'true', -fill=>'both', -side=>'top',);
>
> $mlb->columnInsert('end', -text=>'Test');
>
> # setup the initial indices
> for(1..10000)
> {
> $mlb->insert('end',[$_]);
> }
>
> MainLoop;
>
> sub leak
> {
> $mlb->delete(0,'end');
>
> for(0..10000){
> $mlb->insert($_,[$_]);
> }
>
> $mlb->update;
> }
>
> ########################################
##########
>

x3v0-usenet@yahoo.com

2005-05-01, 9:03 pm

Rob Seegel wrote:
>
> Try the following link, and look at the first four or five threads
> that come up in the List. I haven't looked at the C code for Perl Tk
in
> a while so I can't really say if the patches were somehow
"unpatched",
> never applied, or if there are additional problems present that the
> posted patches didn't cover:
>
>http://groups-beta.google.com/group...coring=d&hl=en&

>
>
> Rob
>

I tested Tk::Listbox itself and it indeed leaks memory. So it seems the
problem isn't MListbox at all. This is a rather large bug in the Tk
distro, I'm surprised it has not been fixed yet.

Rob, you mentioned a patch in your post. Do you know of a patch that I
could apply to Tk::Listbox myself?

- Ken

zentara

2005-05-01, 9:03 pm

On 28 Apr 2005 13:36:22 -0700, x3v0-usenet@yahoo.com wrote:

>This is happening with Tk::MListbox 1.11. I tested it on Fedora Core 2
>and Windows 2000.
>
>I believe this problem has been reported before but noone found a
>solution.
>
>I am observing memory leak when using MListbox. The leak happens both
>when deleting/re-inserting the list and when sorting. I monitored the
>memory leak by using "top" in Linux and using the task manager in
>Win2k.
>
>The memory seems to go up by the same amount every time. I could not
>find the source of this problem. The problem may lie in Tk::Listbox
>itself and not MListbox, I can't say for sure.
>
>The below script demonstrates the problem. Everytime the button is
>clicked or the column is sorted, the script leaks memory.
>

I cannot say for sure, but I (and others) have experienced the same
thing with HList, but HList can be made to work, and I suggest you
use that, if you are updating the list many times.

What happens is that the list keeps an "internal counter" of each new
entry, so when you delete them all, the "internal counter" goes to
1000-2000, etc, even though it shows 1..1000 as the values.

Just as pseudocode, what you need to do is "reuse" those
indices, so instead of deleting 1..1000 and inserting 1..1000 again,
you would "configure" each index to have a new value, like this:

THIS CODE DOES NOT WORK
#!/usr/bin/perl -w
use strict;
use Tk;
use Tk::MListbox;

my $mw = Tk::MainWindow->new( -title => 'MListbox Leak' );

my $b = $mw->Button(
-text => 'Leak',
-command => \&leak
)->pack( -side => 'bottom' );

my $mlb = $mw->Scrolled(
'MListbox',
-scrollbars => 'se',
-selectmode => 'extended',
)->pack( -expand => 'true', -fill => 'both', -side => 'top', );

$mlb->columnInsert( 'end', -text => 'Test' );

# setup the initial indices
for ( 0 .. 1000 ) {
$mlb->insert( $_ , [$_] );
}

MainLoop;

sub leak {
for ( 0 .. 1000 ) {
$mlb->itemconfigure( $_, [$_] );
}
$mlb->update;
}
__END__

Now I looked in Tk::Listbox for itemconfigure, but MListbox(\
which supposedly inherits from Listbox) croaks on itemconfigure.

What that tells me is that Listbox and Mlistbox are NOT meant for
alot of updating, they are "one shot selection lists.

Now HList, will do what you want.

This script uses entryconfigure to reuse indices, which is preferred;
BUT in this simple case, it also works with deleting all then re-adding.
But bear in mind, I have seen where HList will leak too, so always
watch the memory.

#!/usr/bin/perl
use strict;
use Tk;
use Tk::HList;

my $mw = MainWindow->new();

my $hlist = $mw->Scrolled('HList',
-itemtype => 'text',
-selectmode => 'single',
)->pack;

foreach (1..1000) {
my $text = $_.time;
$hlist->add( $_, -text => $text );
}

my $button = $mw->Button(-text => 'Leak Test',
-command => \&testleak)->pack;

MainLoop;
######################################
sub testleak{
#foreach (1..1000) {
# # $hlist->delete('all');
# $hlist->delete('entry',$_);
# }

foreach (1..1000) {
my $text = $_.time;
# $hlist->add( $_, -text => $text );
$hlist->entryconfigure( $_, -text => $text );

}
}
__END__

--
I'm not really a human, but I play one on earth.
http://zentara.net/japh.html

Ken

2005-05-01, 9:03 pm

zentara wrote:
> On 28 Apr 2005 13:36:22 -0700, x3v0-usenet@yahoo.com wrote:
>
> Now I looked in Tk::Listbox for itemconfigure, but MListbox(\
> which supposedly inherits from Listbox) croaks on itemconfigure.
>
> What that tells me is that Listbox and Mlistbox are NOT meant for
> alot of updating, they are "one shot selection lists.
>

I don't think this is true. It seems to me that the leak is simply a bug.

> Now HList, will do what you want.
>

Not exactly what I want. I really miss the fact that HList does not
sort. Perhaps someone should write a MListbox type widget based off of
HList.. Actually, I think I may give that one a try :)

> This script uses entryconfigure to reuse indices, which is preferred;
> BUT in this simple case, it also works with deleting all then re-adding.
> But bear in mind, I have seen where HList will leak too, so always
> watch the memory.
>
> #!/usr/bin/perl
> use strict;
> use Tk;
> use Tk::HList;
>
> my $mw = MainWindow->new();
>
> my $hlist = $mw->Scrolled('HList',
> -itemtype => 'text',
> -selectmode => 'single',
> )->pack;
>
> foreach (1..1000) {
> my $text = $_.time;
> $hlist->add( $_, -text => $text );
> }
>
> my $button = $mw->Button(-text => 'Leak Test',
> -command => \&testleak)->pack;
>
> MainLoop;
> ######################################
> sub testleak{
> #foreach (1..1000) {
> # # $hlist->delete('all');
> # $hlist->delete('entry',$_);
> # }
>
> foreach (1..1000) {
> my $text = $_.time;
> # $hlist->add( $_, -text => $text );
> $hlist->entryconfigure( $_, -text => $text );
>
> }
> }
> __END__
>
>

Thanks, I'm definately going to try out HList.

- Ken

Rob Seegel

2005-05-01, 9:03 pm

zentara wrote:
> On 28 Apr 2005 13:36:22 -0700, x3v0-usenet@yahoo.com wrote:
>
>
> I cannot say for sure, but I (and others) have experienced the same
> thing with HList, but HList can be made to work, and I suggest you
> use that, if you are updating the list many times.

It is true that some of the memory problems can be mitigated with HList,
however this is really a workaround, and not without its own costs in
terms of complexity and in terms of the amount of code that needs to
written. Besides, the idea of reconfiguring verses creating new entries
is an optimization that ought to be occuring inside the C code and not
in the perl code. This should be transparant.

> What happens is that the list keeps an "internal counter" of each new
> entry, so when you delete them all, the "internal counter" goes to
> 1000-2000, etc, even though it shows 1..1000 as the values.

If this is truly the only problem, then a "fix" would be to reset the
internal counter each time there was a delete operation that resulted in
no more rows. This could conceivably work with both Widgets -- I
wonder if that truly is the only problem. I doubt it. By itself, not
resetting the internal counter doesn't seem to me like it would result
in a radical memory problems (unless the other entries indexed at those
lower ranges were never freed.) Internally, there is an array that
includes each entry. The c code only "paints" those entries that should
be visible at a given moment. This is based off the last time I took a
p

at the c code, which I haven't done since the 804 changes.

In the latest releases, both HList and Listbox widgets can have other
objects that get created such as "styles". Personally, I'd be interested
in knowing if the problem was still present in the pre Tk804.* releases.
If not, and your application doesn't require newer functionality found
in Tk804 then you might consider using an older version until the
problem can be resolved instead of using a workaround.

> Now I looked in Tk::Listbox for itemconfigure, but MListbox(\
> which supposedly inherits from Listbox) croaks on itemconfigure.

A subtle but important distinction here: MListbox is not a subclass of
Listbox - it doesn't inherit anything from Listbox (except for memory
leaks, apparently ;-)). MListbox is a composite -- it is composed of
other widgets. It has MLColumn Widgets, which are composed of HButton
and CListbox widgets. CListbox __IS__ derived from Listbox, and WILL
support itemconfigure or any other method that Listbox currently
provides. The code doesn't prevent this from happening in any way. In
fact, MLColumn delegates most of it's method calls to CListbox.

This is nitpicking, but it is true that MListbox doesn't currently
provide any delegation methods to support the enhanced functionality
that Listbox provides. I'll take full blame for that - I haven't pushed
a new version out in a while, and I hadn't decided how I would support
the enhancements when a sort was triggered.

> What that tells me is that Listbox and Mlistbox are NOT meant for
> alot of updating, they are "one shot selection lists.

What it tells me is that there is buggy behavior that stands in the way
of lots of updating, and that HList just happens to have a way of
working around it. I have a hard time believing that was the primary
reason that the configuration methods for HList were created. You can
also get in trouble with HList when deleting rows, then recreating them.
I don't mean to criticise the approach, but let's call it for what it is
(a workaround) -- I've used the same technique for HList in the past.

Let me describe some of the issue's I ran into when I created a fairly
signifigant (and unfortunately proprietary) widget back in 2001.
I wrote an admin app that was a front end to a database. This predated
the TableMatix widget, and it also predated the 804 releases. I needed
the capabilty to have resizable columns, sorting, and be able to use
different foreground colors or font styles depending on content in a
row. In addition, I thought it would be nice if I could have a two-tone
background scheme for the displayed rows.

The only widget that provided the base capabiliities at the time was
HList, but it was immediately clear that I had to do a lot of work
before I'd have a widget that did all of what I needed.

For performance reasons, I found it more efficient to reconfigure rather
than recreate new rows. I only created rows when there were not enough
to display all the data that I needed to show, and I deleted rows when
there were too many. Otherwise I reconfigured existing rows. Given the
characteristics of HList, and I can see now that it would still have
leaked memory, but it would have done it much slower because of the
reconfiguration.

The resulting widget did all the things that I'd wanted it to do and
more, and it did it in a way that I could reuse (which was VERY
important for my application).

There were still issues, however:

PERFORMANCE:

- It was slow. MListbox, Columns, and TableMatrix will always win over
HList if you are interested in pure performance. Why was this?

Imagine I have a listbox with 1000 elements. Compare the relative speeds
of the following snippets:

$listbox->delete(0, 'end');

or

foreach (1 .. 1000) { $listbox->delete(0); }

Guess which is going to perform better ;-)

As I mentioned above, I used configure over re create with HList because
it was fewer calls to reconfigure something than it was to create
something, and it seemed like common sense that reusing an existing
object would be more efficient than going to the trouble to create one.
This turned out to be true.

In a sense, programming with Tk is a lot like network programming
especially when you are dealing with lots of data. Sending batches of
data from the perl code to the c code is much more efficient than having
an individual call for

Let's say I have 1000 elements in an HList and 1000 elements within a
one-column MListbox. In HList, I'm using the default style, and not
creating any new ones. In both cases I have the contents of each List
stored in an array for convenience of the example:

If I sort the array and modifiy the display for HList, I'm going to have
to have to make 1000 calls. If I do the same thing for MListbox, there
are basically two calls.

Now imagine there are 10 columns each containing 1000 rows. This
translates into 10000 calls for HList and 20 calls for MListbox/Columns.

In order to get past the performance gap, my widget had to have an
additional option called -updaterate that allowed a number to be
specified. Every x number of rows that were updated within HList, I
updated the display so that it wouldn't look like it was hanging too badly.

COMPLEXITY:

I found that the code within my resulting Widget to be much more
complex than I thought it should be, and it really rankled me that I was
doing some things that should have been much easier or were workarounds
to account for interface differences between a Tix widget HList, and a
core Tk widget - Listbox, which were both in the base distro. It was a
good learning experience, but VERY frustrating.

Buggy HList behavior:

Of all the widgets with the Perl/Tk distro, I find that the Tix derived
widgets to have the worst event handlers. The selection modes are
inconsistant with the Tk ones, and the way anchors were handled was
inconsistent with the way in which Listbox did. Some of these complaints
may have been fixed, but at the time they weren't and so I had to
rewrite many of them so that a single button press and release was not
calling the same callback more than once. Sometimes many more times than
once.

I can understand why the Tix widgets were brought in - they provided
behavior that simply wasn't available in Tk, but I think it would have
been if they had been reworked more to make them more consistent with
the base Tk widgets.

MISCELLANEOUS PROBLEMS:

Header problems:

You cannot bind events to the HList headers, so you are forced to embed
widgets in them, and bind to those widgets instead. I wanted to be able
to resize my columns, so I borrowed a trick from Tk::Columns that
featured a decoration within each header that could be used to resize
the column. I modified my header button to draw a line (floating frame)
down the face of the column to illustrate where the new column boundary
would be when I released the mouse button.

Also, you determine the number of Headers that the widget supports at
creation time, and that number can be modified without recreating the
HList -- fortunately, this rarely was an issue, but that was exactly the
way I dealt with it within my widget.

Embedded Widgets/Images

Mostly the problem is with Widgets. WHen you embed widgets within a cell
in HList you can see it overwritting the 3D "border" I mitigated this by
putting HList in another frame, and using the Frame for my "border", and
then setting the HList border to 0. Incidentally, you will see this same
problem in Text and Canvas I believe.

I also played around with using a separate set of widgets entirely for
the Header, and didn't use the header functionality provided by HList.
This had it's own share of pluses and minuses. For the most part I liked
the results better.

Rob

zentara

2005-05-01, 9:03 pm

On Sat, 30 Apr 2005 19:00:18 -0400, Rob Seegel <RobSeegel@comcast.net>
wrote:

>zentara wrote:
[color=darkred]
>
>If this is truly the only problem, then a "fix" would be to reset the
>internal counter each time there was a delete operation that resulted in
> no more rows. This could conceivably work with both Widgets -- I
>wonder if that truly is the only problem. I doubt it. By itself, not
>resetting the internal counter doesn't seem to me like it would result
>in a radical memory problems (unless the other entries indexed at those
> lower ranges were never freed.) Internally, there is an array that
>includes each entry. The c code only "paints" those entries that should
>be visible at a given moment. This is based off the last time I took a
>p

at the c code, which I haven't done since the 804 changes.

Thanks for the detailed analysis of the problem. I didn't mean to
disparage MListbox or Listbox, I was just pointing out the
"practicalities" of using them for contantly changing entries.
A few K increase in memory is acceptable if the program is not
running for long periods.

I have no direct knowledge of the "internal counter", except from my
playing around with HList, when I encountered this same problem,
and was running DataDumper on it, and found that after deleting entries,
and adding new ones, the Dump showed remnants of old entries laying
around, with a hidden count which kept going up.
I didn't dig into it, I just worked around it with "reconfiguring
entries".
Maybe it has something to do with the "ref count" Perl keeps of internal
variables?
I wish I could find the old Dumper scripts I used when testing it. ;-(

--
I'm not really a human, but I play one on earth.
http://zentara.net/japh.html

Rob Seegel

2005-05-02, 3:58 am

zentara wrote:
> On Sat, 30 Apr 2005 19:00:18 -0400, Rob Seegel <RobSeegel@comcast.net>
> wrote:

>
> Thanks for the detailed analysis of the problem. I didn't mean to
> disparage MListbox or Listbox, I was just pointing out the
> "practicalities" of using them for contantly changing entries.
> A few K increase in memory is acceptable if the program is not
> running for long periods.

I know you weren't - you were just trying to provide alternatives. Sorry
for the strong reaction. Really, my main point (as rambling as it was)
was that although HList can be used, its not exactly a "free" lunch.
Depending on the functionality someone needs it's going to require a
little work, and even then it may or may not live up to expectations.

It is a pretty big deal that a core widget such as Listbox has these
problems and I wish that we could focus more on finding the problem. It
seems to me like this is a fairly new issue -- I don't recall a lot of
complaints with the older code. It makes me wonder:

- Does the same problem occur with Tcl/Tk?

I suspect probably not, because it is likely if it did it would already
have been discovered and fixed. Looking back at the patch, and some of
the comments that were made, it may have something to do with reference
counting for eventual gc, and the fact that for Perl and Tcl may have
minor differences in the way it's handled. Pure speculation...

- Which release broke it?

This might be handy for going back to a previous release, where it
wasn't broken.

> I have no direct knowledge of the "internal counter", except from my
> playing around with HList, when I encountered this same problem,
> and was running DataDumper on it, and found that after deleting entries,
> and adding new ones, the Dump showed remnants of old entries laying
> around, with a hidden count which kept going up.

I was just glancing at the source code, and it looks like although there
are a few similiarities, Listbox code is quite a bit different from the
HList code (which is what I would expect), and doesn't look like it has
an internal counter used for assigning identifiers. If there is a
counter then it more likely to do with reference counting mentioned earlier.

> I didn't dig into it, I just worked around it with "reconfiguring
> entries".
> Maybe it has something to do with the "ref count" Perl keeps of internal
> variables?
> I wish I could find the old Dumper scripts I used when testing it. ;-(
>
>

Rob

Sponsored Links