Error allocating multiple buffers over 1GB

I am allocating 6 large unsigned char * buffers in quick succession. I am
using the new operator to allocate the memory. The size of the buffers
changes based on the application's settings. If the total memory needed for
all the buffers exceeds 1 GB, when I try to allocate the buffer that will
exceed this apparent 1GB threshold, the call to new returns null and
GetLastError() returns 8, indicating there is not enough storage. So, for
example, if each buffer is supposed to be 180 MB, when trying to allocate th
e
6th buffer, the memory allocation fails. This is occurring on a machine with
3 GB of memory. When I bring up Task Manager at the point of failure, it
indicates that the application's process Mem usage is about 1.3 GB, and ther
e
is still over 1.3 GB of physical memory available. Does anybody know why I
can't allocate all the buffers?

TIA
Eric

"Eric Wade" <ewade10@hotmail.com.(donotspam)> wrote in message
news:6A984808-BF10-4E28-8A91-27EA56E50E5C@microsoft.com...
>I am allocating 6 large unsigned char * buffers in quick succession. I am
> using the new operator to allocate the memory. The size of the buffers
> changes based on the application's settings. If the total memory needed
> for
> all the buffers exceeds 1 GB, when I try to allocate the buffer that will
> exceed this apparent 1GB threshold, the call to new returns null and
> GetLastError() returns 8, indicating there is not enough storage. So, for
> example, if each buffer is supposed to be 180 MB, when trying to allocate
> the
> 6th buffer, the memory allocation fails. This is occurring on a machine
> with
> 3 GB of memory. When I bring up Task Manager at the point of failure, it
> indicates that the application's process Mem usage is about 1.3 GB, and
> there
> is still over 1.3 GB of physical memory available. Does anybody know why I
> can't allocate all the buffers?

There are a couple of things that you have to consider.

1) In general, (but not always) you only have 2GB of virtual address space
available (2GB reserved for the o/s)
2) The virtual address space that you have may be fragmented without a
sufficiently large contiguous chunk

Regards,
Will

On Thu, 10 Mar 2005 11:03:03 -0800, Eric Wade
<ewade10@hotmail.com.(donotspam)> wrote:

>I am allocating 6 large unsigned char * buffers in quick succession. I am
>using the new operator to allocate the memory. The size of the buffers
>changes based on the application's settings. If the total memory needed for
>all the buffers exceeds 1 GB, when I try to allocate the buffer that will
>exceed this apparent 1GB threshold, the call to new returns null and
>GetLastError() returns 8, indicating there is not enough storage. So, for
>example, if each buffer is supposed to be 180 MB, when trying to allocate t
he
>6th buffer, the memory allocation fails. This is occurring on a machine wit
h
>3 GB of memory. When I bring up Task Manager at the point of failure, it
>indicates that the application's process Mem usage is about 1.3 GB, and the
re
>is still over 1.3 GB of physical memory available. Does anybody know why I
>can't allocate all the buffers?

A single process's maximum memory, for code, stacks, data and
miscellaneous stuff is 2GB. When you 6th allocation fails, there is
not 180Mb of contiguous memory available.

Here are several possible solutions:

1. Make your buffers smaller and manage them differently.
2. Use a memory mapped file and map 180MB windows into it as
necessary.
3. Use AWD (Address Windowing Extensions)
4. Add /3GB in BOOT.INI, which increases a process's limit to 3GB

Use MSDN to find out more about 3 & 4. I'm not sure which O/S versions
they are available on.


--
Sev

"William DePalo [MVP VC++]" wrote:

> "Eric Wade" <ewade10@hotmail.com.(donotspam)> wrote in message
> news:6A984808-BF10-4E28-8A91-27EA56E50E5C@microsoft.com... 
>
> There are a couple of things that you have to consider.
>
> 1) In general, (but not always) you only have 2GB of virtual address space
> available (2GB reserved for the o/s)
> 2) The virtual address space that you have may be fragmented without a
> sufficiently large contiguous chunk
>
> Regards,
> Will
>
>
>
>

I've had that suggested to me, but it implies that the error would occur
randomly based on the level of fragmentation of the memory. However, it is
very consistent. It always happens just above the 1GB limit. I also get the
exact same behavior on a machine with only 2GB of memory, where you would
expect the fragmentation to be worse, and the error to occur sooner, but it
doesn't.

Eric

"Eric Wade" <ewade10@hotmail.com.(donotspam)> wrote in message
news:1B8532C7-A30F-497F-ADDC-6E39C3685742@microsoft.com...
> "William DePalo [MVP VC++]" wrote:
... 
...
>
> I've had that suggested to me, but it implies that the error would occur
> randomly based on the level of fragmentation of the memory. However, it is
> very consistent. It always happens just above the 1GB limit. I also get th
e
> exact same behavior on a machine with only 2GB of memory, where you would
> expect the fragmentation to be worse, and the error to occur sooner, but i
t
> doesn't.

Your response indicates that you do not yet distinguish
between physical memory and virtual memory, nor
between physical addresses and the logical addresses
presented to each running process.  You may want to
read up on those concepts someday, although it is not
likely to help cure your present problem.

--
--Larry Brasfield
email: donotspam_larry_brasfield@hotmail.com
Above views may belong only to me.

"Eric Wade" <ewade10@hotmail.com.(donotspam)> wrote in message
news:1B8532C7-A30F-497F-ADDC-6E39C3685742@microsoft.com...
> I've had that suggested to me, but it implies that the error would occur
> randomly based on the level of fragmentation of the memory.

Hmm, we are talking about virtual memory here, right? The amount of physical
memory that you have may de dependent on what all else is ongoing on that
box, but if all the DLLs you load have not been rebuilt between iterations
and if the heap gets allocated at the same spot in virtual memory then you
should see some consistency.

> I also get the exact same behavior on a machine with only 2GB
> of memory, where you would expect the fragmentation to be
> worse, and the error to occur sooner, but it doesn't.

That data point fits perfectly with the virtual memory fragmentation
hypothesis.

Someone may pop in here and offer the name of a utility which can
dynamically display your process' memory layout for you. If no one does, you
might want to check the docs for EnumProcessModules() and
GetMoudleInformation(). They should allow you to get information on DLL
memory usage. And get ProcessHeap() and HeapWalk() should get you some
information on the memory occupied by your heaps. The information they
provide should be able to confirm or dismiss the virtual memory
fragmentation hypothesis.

Regards,
Will

"Severian" wrote:

> On Thu, 10 Mar 2005 11:03:03 -0800, Eric Wade
> <ewade10@hotmail.com.(donotspam)> wrote:
> 
>
> A single process's maximum memory, for code, stacks, data and
> miscellaneous stuff is 2GB. When you 6th allocation fails, there is
> not 180Mb of contiguous memory available.
>
> Here are several possible solutions:
>
> 1. Make your buffers smaller and manage them differently.
> 2. Use a memory mapped file and map 180MB windows into it as
> necessary.
> 3. Use AWD (Address Windowing Extensions)
> 4. Add /3GB in BOOT.INI, which increases a process's limit to 3GB
>
> Use MSDN to find out more about 3 & 4. I'm not sure which O/S versions
> they are available on.
>
>
> --
> Sev
>

Task Manager indicates that the Mem Usage of the application's process is
only 1.3 GB. On the surface, this seems to indicate that there should be 700
MB of unused memory for the application to use. Is there some way to
programmatically verify this? Is there some way to check the size of the
largest chunck of contiguous memory available?

In regards to 1: I do have some fall back code in place to do that. However,
the buffers are large to improve performance when the data is written to
disk. Writing more small buffers is hurting the application's performance.

In regards to 2: The buffers are large and in memory for performance
reasons. The additional disk access would hurt performance.

In regards to 3: I'm not familiar with AWD. I will take a look at it.

In regards to 4: I tried setting the /LARGEADDRESSAWARE flag when linking,
expecting that to extend the process memory to 3 GB, but it didn't help. Am
I misunderstanding
something? Or, is that only part of it, and setting /3GB in boot.ini is what
makes it work?

Thanks,
Eric

On Thu, 10 Mar 2005 13:41:05 -0800, Eric Wade
<ewade10@hotmail.com.(donotspam)> wrote:

>
>
>"Severian" wrote:
> 
>
>Task Manager indicates that the Mem Usage of the application's process is
>only 1.3 GB. On the surface, this seems to indicate that there should be 70
0
>MB of unused memory for the application to use. Is there some way to
>programmatically verify this? Is there some way to check the size of the
>largest chunck of contiguous memory available?

You could probably create one using VirtualQueryEx(), but it would be
a mess.

>In regards to 1: I do have some fall back code in place to do that. However
,
>the buffers are large to improve performance when the data is written to
>disk. Writing more small buffers is hurting the application's performance.

You might want to look at WriteFileGather().

Also, use VirtualAlloc() for your large allocations to avoid heap
overhead and any possible fragmentation caused by it.

>In regards to 2: The buffers are large and in memory for performance
>reasons. The additional disk access would hurt performance.

Memory mapped file data is not actually written to disk unless
requested or necessary (low system memory, etc.) Think of it as a
"private" page file.

>In regards to 3: I'm not familiar with AWD. I will take a look at it.
>
>In regards to 4: I tried setting the /LARGEADDRESSAWARE flag when linking,
>expecting that to extend the process memory to 3 GB, but it didn't help. Am
>I misunderstanding
>something? Or, is that only part of it, and setting /3GB in boot.ini is wha
t
>makes it work?

Yes, I believe /3GB is required for /LARGEADRESSAWARE to take effect
in the executable, but I've never used it so I'm not sure.

Also, is this some special disk? I can't imagine writing 10 18MB
buffers taking significantly longer than 1 180MB buffer.

Are you using the appropriate flags when you open the file?
(FILE_FLAG_...) You may find that using one or more of these improves
performance.

--
Sev

> Someone may pop in here and offer the name of a utility which can
> dynamically display your process' memory layout for you.

the `!address` debugger extension command in cdb/ntsd/windbg
displays the layout of the address space of your process,
inluding the largest contiguous availalble region.

Heap fragmentation can be inferred by `!heap -s`.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm


"William DePalo [MVP VC++]" <willd.no.spam@mvps.org> wrote in message
news:OBtIsQbJFHA.2764@tk2msftngp13.phx.gbl...
> "Eric Wade" <ewade10@hotmail.com.(donotspam)> wrote in message
> news:1B8532C7-A30F-497F-ADDC-6E39C3685742@microsoft.com... 
>
> Hmm, we are talking about virtual memory here, right? The amount of
physical
> memory that you have may de dependent on what all else is ongoing on that
> box, but if all the DLLs you load have not been rebuilt between iterations
> and if the heap gets allocated at the same spot in virtual memory then you
> should see some consistency.
> 
>
> That data point fits perfectly with the virtual memory fragmentation
> hypothesis.
>
> Someone may pop in here and offer the name of a utility which can
> dynamically display your process' memory layout for you. If no one does,
you
> might want to check the docs for EnumProcessModules() and
> GetMoudleInformation(). They should allow you to get information on DLL
> memory usage. And get ProcessHeap() and HeapWalk() should get you some
> information on the memory occupied by your heaps. The information they
> provide should be able to confirm or dismiss the virtual memory
> fragmentation hypothesis.
>
> Regards,
> Will
>
>

Note that a single I/O operation in Windows cannot exceed 64 MB... (unless
it's very suboptimal cached I/O, causing severe cache bloat).

"Severian" <severian@chlamydia-is-not-a-flower.com> wrote in message
 news:1ih1315e9q9m1qtbavrp4c8qagnu17n68m@
4ax.com...
> On Thu, 10 Mar 2005 13:41:05 -0800, Eric Wade
> <ewade10@hotmail.com.(donotspam)> wrote:
> 
>
> Also, is this some special disk? I can't imagine writing 10 18MB
> buffers taking significantly longer than 1 180MB buffer.
>
> Are you using the appropriate flags when you open the file?
> (FILE_FLAG_...) You may find that using one or more of these improves
> performance.
>
> --
> Sev