Mathematica - memory issues

Author

memory issues

Christoph Lhotka

2006-08-18, 4:07 am

Hi!

Running Mathematica 5.2 on 64Bit (AMD) with 2GB of RAM Memory OS: WindowsXP 64
Pro. I get the message: No more memory available. Mathematica Kernel has shut
down...

How do I find out:

1) How much memory can Mathematica store in the RAM?
2) How much memory can Mathematica store on the Hard disk (swap file)?
3) At which memory limit does Mathematica begin writting to the had disk?
3) Can I influence these limit 3) using the OS or in Mathematica?
4) How do I find out at which point in the calculation the kernel stops?
5) Why does my dual core processor does not work higher than with 50%

Doing some tests on my own I got the following riddle:

In[]:=ByteCount[1]
Out[]:=16

In[]:=2000000000/16
Out[]:=125000000

In[]:=DAT=Table[i,{i,1,125000000}];

In[]:=ByteCount[DAT]
Out[]:=500000056

In[]:=MaxMemoryUsed[]
Out[]:= 502133312

Why is a list of 125 000 000 different integers (a 16 Byte) only 500 000 000
Bytes long? How should I estimate the memory limits in my algorithms, when
designing them on this basis?

Thank you in advance

Christoph

Virgilio, Vincent - SSD

2006-08-19, 4:03 am

Perhaps this will help answer your last question:

In[1]:=

m=Table[ByteCount[Range[i]],{i,10}]
Rest[m]-Most[m]

Out[1]=

{60,64,68,72,76,80,84,88,92,96}

Out[2]=

{4,4,4,4,4,4,4,4,4}

Each exact integer added to the list (from Range) uses an additional 4
bytes. Then, 4 * 125*^6 = 500*^6, which is about the memory footprint
you note for DAT below.

Vince Virgilio

(However, I notice ByteCount[{1}] != ByteCount[Range[1], i.e. 40 != 60.
ByteCount does not have Hold attributes, so it shouldn't see
Range[1]---does it somehow?)

-----Original Message-----
From: Christoph Lhotka [mailto:lhotka@astro.univie.ac.at]
Subject: memory issues

Hi!

Running Mathematica 5.2 on 64Bit (AMD) with 2GB of RAM Memory OS:
WindowsXP 64 Pro. I get the message: No more memory available.
Mathematica Kernel has shut down...

How do I find out:

1) How much memory can Mathematica store in the RAM?
2) How much memory can Mathematica store on the Hard disk (swap file)?
3) At which memory limit does Mathematica begin writting to the had
disk?
3) Can I influence these limit 3) using the OS or in Mathematica?
4) How do I find out at which point in the calculation the kernel stops?
5) Why does my dual core processor does not work higher than with 50%

Doing some tests on my own I got the following riddle:

In[]:=ByteCount[1]
Out[]:=16

In[]:=2000000000/16
Out[]:=125000000

In[]:=DAT=Table[i,{i,1,125000000}];

In[]:=ByteCount[DAT]
Out[]:=500000056

In[]:=MaxMemoryUsed[]
Out[]:= 502133312

Why is a list of 125 000 000 different integers (a 16 Byte) only 500 000
000 Bytes long? How should I estimate the memory limits in my
algorithms, when designing them on this basis?

Thank you in advance

Christoph

************************************
This e-mail and any files transmitted with it are proprietary and intended solely
for the use of the individual or entity to whom they are addressed. If you have
received this e-mail in error please notify the sender. Please note that any views
or opinions presented in this e-mail are solely those of the author and do not
necessarily represent those of ITT, Inc. The recipient should check
this e-mail and any attachments for the presence of viruses. ITT accepts
no liability for any damage caused by any virus transmitted by this e-mail.
************************************

Jean-Marc Gulliet

2006-08-19, 4:03 am

Christoph Lhotka wrote:
> Hi!
>
> Running Mathematica 5.2 on 64Bit (AMD) with 2GB of RAM Memory OS: WindowsXP 64
> Pro. I get the message: No more memory available. Mathematica Kernel has shut
> down...
>
> How do I find out:
> [ ... ]
> 5) Why does my dual core processor does not work higher than with 50%

The thread "Illusory Multicore Support in 5.2?", April 2006, should
answer your question,
http://forums.wolfram.com/mathgroup...r/msg00673.html

>
> Doing some tests on my own I got the following riddle:
>
> In[]:=ByteCount[1]
> Out[]:=16
>
> In[]:=2000000000/16
> Out[]:=125000000
>
> In[]:=DAT=Table[i,{i,1,125000000}];
>
> In[]:=ByteCount[DAT]
> Out[]:=500000056
>
> In[]:=MaxMemoryUsed[]
> Out[]:= 502133312
>
> Why is a list of 125 000 000 different integers (a 16 Byte) only 500 000 000
> Bytes long? How should I estimate the memory limits in my algorithms, when
> designing them on this basis?

Mathematica uses packed arrays: "When appropriate, large lists and
nested lists of numbers are automatically stored as packed arrays of
machine-sized integers or real numbers [1], [2]."

Regards,
Jean-Marc

[1] "A.9.2 Data Structures and Memory Management"
http://documents.wolfram.com/mathem...k/section-A.9.2

[2] "Packed Array Summary"
http://documents.wolfram.com/mathem...gebra5.0.2.html

Jens-Peer Kuska

2006-08-19, 4:03 am

Hi,

only a 32 bit operating system has the limit, that it can only address
4 GByte. But typical the operating system has some overhead to handle
multiple applications ...

Especial Windows XP 32 allow a bit less than 2 GByte per application.

With a 64 operating system you can address 4 TByte ... but your 2 GByte
RAM are far away from that limit.

Mathematica can store so many data as the operating system allows,
the operating system uses virtual memory and for an application it
does not make a difference if the memory is in the RAM or in th swap
file. It will only take longer time to handle the data from the swap
file. Mathematica can't find out when the operating system starts using
the swap file. Mathematica write *never* memory in the swap file -- your
operating system does this ...

Mathematica uses OpenMP to perform certain numerical computation in
parallel and you have to set the environment variable OMP_NUM_THREADS=4
or what ever to tell Mathematica how many threads can be started parallel.

Mathematica store integers as machine integers whenever it can.
A machine integer on a 32 bit CPU can be 2^31 -1 (one bit for the sign),
and that needs 4 bytes not 16, on a 64 bit CPU you can have long
integers that store values up to 2^63-1 and these numbers need 8 byte
and never 16. And in you example you have obvious 32 bit machine
integers because
4 (bytes per integer)* 125000000==500000000
and you have a small overhead of 56 byte to store the array ...

And
> How should I estimate the memory limits in my algorithms, when
> designing them on this basis?

it seems the best to take a break and to think about some basic
properties of the computer you are using and to read the manual of your
operating system.

Regards
Jens

Christoph Lhotka wrote:
> Hi!
>
> Running Mathematica 5.2 on 64Bit (AMD) with 2GB of RAM Memory OS: WindowsXP 64
> Pro. I get the message: No more memory available. Mathematica Kernel has shut
> down...
>
> How do I find out:
>
> 1) How much memory can Mathematica store in the RAM?
> 2) How much memory can Mathematica store on the Hard disk (swap file)?
> 3) At which memory limit does Mathematica begin writting to the had disk?
> 3) Can I influence these limit 3) using the OS or in Mathematica?
> 4) How do I find out at which point in the calculation the kernel stops?
> 5) Why does my dual core processor does not work higher than with 50%
>
> Doing some tests on my own I got the following riddle:
>
> In[]:=ByteCount[1]
> Out[]:=16
>
> In[]:=2000000000/16
> Out[]:=125000000
>
> In[]:=DAT=Table[i,{i,1,125000000}];
>
> In[]:=ByteCount[DAT]
> Out[]:=500000056
>
> In[]:=MaxMemoryUsed[]
> Out[]:= 502133312
>
> Why is a list of 125 000 000 different integers (a 16 Byte) only 500 000 000
> Bytes long? How should I estimate the memory limits in my algorithms, when
> designing them on this basis?
>
> Thank you in advance
>
> Christoph
>

Sponsored Links