Home > Archive > VC Language > March 2005 > Error allocating multiple buffers over 1GB
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Error allocating multiple buffers over 1GB
|
|
| Eric Wade 2005-03-10, 9:00 pm |
| I am allocating 6 large unsigned char * buffers in quick succession. I am
using the new operator to allocate the memory. The size of the buffers
changes based on the application's settings. If the total memory needed for
all the buffers exceeds 1 GB, when I try to allocate the buffer that will
exceed this apparent 1GB threshold, the call to new returns null and
GetLastError() returns 8, indicating there is not enough storage. So, for
example, if each buffer is supposed to be 180 MB, when trying to allocate the
6th buffer, the memory allocation fails. This is occurring on a machine with
3 GB of memory. When I bring up Task Manager at the point of failure, it
indicates that the application's process Mem usage is about 1.3 GB, and there
is still over 1.3 GB of physical memory available. Does anybody know why I
can't allocate all the buffers?
TIA
Eric
| |
| William DePalo [MVP VC++] 2005-03-10, 9:00 pm |
| "Eric Wade" <ewade10@hotmail.com.(donotspam)> wrote in message
news:6A984808-BF10-4E28-8A91-27EA56E50E5C@microsoft.com...
>I am allocating 6 large unsigned char * buffers in quick succession. I am
> using the new operator to allocate the memory. The size of the buffers
> changes based on the application's settings. If the total memory needed
> for
> all the buffers exceeds 1 GB, when I try to allocate the buffer that will
> exceed this apparent 1GB threshold, the call to new returns null and
> GetLastError() returns 8, indicating there is not enough storage. So, for
> example, if each buffer is supposed to be 180 MB, when trying to allocate
> the
> 6th buffer, the memory allocation fails. This is occurring on a machine
> with
> 3 GB of memory. When I bring up Task Manager at the point of failure, it
> indicates that the application's process Mem usage is about 1.3 GB, and
> there
> is still over 1.3 GB of physical memory available. Does anybody know why I
> can't allocate all the buffers?
There are a couple of things that you have to consider.
1) In general, (but not always) you only have 2GB of virtual address space
available (2GB reserved for the o/s)
2) The virtual address space that you have may be fragmented without a
sufficiently large contiguous chunk
Regards,
Will
| |
| Severian 2005-03-10, 9:00 pm |
| On Thu, 10 Mar 2005 11:03:03 -0800, Eric Wade
<ewade10@hotmail.com.(donotspam)> wrote:
>I am allocating 6 large unsigned char * buffers in quick succession. I am
>using the new operator to allocate the memory. The size of the buffers
>changes based on the application's settings. If the total memory needed for
>all the buffers exceeds 1 GB, when I try to allocate the buffer that will
>exceed this apparent 1GB threshold, the call to new returns null and
>GetLastError() returns 8, indicating there is not enough storage. So, for
>example, if each buffer is supposed to be 180 MB, when trying to allocate the
>6th buffer, the memory allocation fails. This is occurring on a machine with
>3 GB of memory. When I bring up Task Manager at the point of failure, it
>indicates that the application's process Mem usage is about 1.3 GB, and there
>is still over 1.3 GB of physical memory available. Does anybody know why I
>can't allocate all the buffers?
A single process's maximum memory, for code, stacks, data and
miscellaneous stuff is 2GB. When you 6th allocation fails, there is
not 180Mb of contiguous memory available.
Here are several possible solutions:
1. Make your buffers smaller and manage them differently.
2. Use a memory mapped file and map 180MB windows into it as
necessary.
3. Use AWD (Address Windowing Extensions)
4. Add /3GB in BOOT.INI, which increases a process's limit to 3GB
Use MSDN to find out more about 3 & 4. I'm not sure which O/S versions
they are available on.
--
Sev
| |
| Eric Wade 2005-03-10, 9:00 pm |
| "William DePalo [MVP VC++]" wrote:
> "Eric Wade" <ewade10@hotmail.com.(donotspam)> wrote in message
> news:6A984808-BF10-4E28-8A91-27EA56E50E5C@microsoft.com...
>
> There are a couple of things that you have to consider.
>
> 1) In general, (but not always) you only have 2GB of virtual address space
> available (2GB reserved for the o/s)
> 2) The virtual address space that you have may be fragmented without a
> sufficiently large contiguous chunk
>
> Regards,
> Will
>
>
>
>
I've had that suggested to me, but it implies that the error would occur
randomly based on the level of fragmentation of the memory. However, it is
very consistent. It always happens just above the 1GB limit. I also get the
exact same behavior on a machine with only 2GB of memory, where you would
expect the fragmentation to be worse, and the error to occur sooner, but it
doesn't.
Eric
| |
| Larry Brasfield 2005-03-10, 9:00 pm |
| "Eric Wade" <ewade10@hotmail.com.(donotspam)> wrote in message
news:1B8532C7-A30F-497F-ADDC-6E39C3685742@microsoft.com...
> "William DePalo [MVP VC++]" wrote:
....
....[color=darkred]
>
> I've had that suggested to me, but it implies that the error would occur
> randomly based on the level of fragmentation of the memory. However, it is
> very consistent. It always happens just above the 1GB limit. I also get the
> exact same behavior on a machine with only 2GB of memory, where you would
> expect the fragmentation to be worse, and the error to occur sooner, but it
> doesn't.
Your response indicates that you do not yet distinguish
between physical memory and virtual memory, nor
between physical addresses and the logical addresses
presented to each running process. You may want to
read up on those concepts someday, although it is not
likely to help cure your present problem.
--
--Larry Brasfield
email: donotspam_larry_brasfield@hotmail.com
Above views may belong only to me.
| |
| William DePalo [MVP VC++] 2005-03-10, 9:00 pm |
| "Eric Wade" <ewade10@hotmail.com.(donotspam)> wrote in message
news:1B8532C7-A30F-497F-ADDC-6E39C3685742@microsoft.com...
> I've had that suggested to me, but it implies that the error would occur
> randomly based on the level of fragmentation of the memory.
Hmm, we are talking about virtual memory here, right? The amount of physical
memory that you have may de dependent on what all else is ongoing on that
box, but if all the DLLs you load have not been rebuilt between iterations
and if the heap gets allocated at the same spot in virtual memory then you
should see some consistency.
> I also get the exact same behavior on a machine with only 2GB
> of memory, where you would expect the fragmentation to be
> worse, and the error to occur sooner, but it doesn't.
That data point fits perfectly with the virtual memory fragmentation
hypothesis.
Someone may pop in here and offer the name of a utility which can
dynamically display your process' memory layout for you. If no one does, you
might want to check the docs for EnumProcessModules() and
GetMoudleInformation(). They should allow you to get information on DLL
memory usage. And get ProcessHeap() and HeapWalk() should get you some
information on the memory occupied by your heaps. The information they
provide should be able to confirm or dismiss the virtual memory
fragmentation hypothesis.
Regards,
Will
| |
| Eric Wade 2005-03-10, 9:00 pm |
|
"Severian" wrote:
> On Thu, 10 Mar 2005 11:03:03 -0800, Eric Wade
> <ewade10@hotmail.com.(donotspam)> wrote:
>
>
> A single process's maximum memory, for code, stacks, data and
> miscellaneous stuff is 2GB. When you 6th allocation fails, there is
> not 180Mb of contiguous memory available.
>
> Here are several possible solutions:
>
> 1. Make your buffers smaller and manage them differently.
> 2. Use a memory mapped file and map 180MB windows into it as
> necessary.
> 3. Use AWD (Address Windowing Extensions)
> 4. Add /3GB in BOOT.INI, which increases a process's limit to 3GB
>
> Use MSDN to find out more about 3 & 4. I'm not sure which O/S versions
> they are available on.
>
>
> --
> Sev
>
Task Manager indicates that the Mem Usage of the application's process is
only 1.3 GB. On the surface, this seems to indicate that there should be 700
MB of unused memory for the application to use. Is there some way to
programmatically verify this? Is there some way to check the size of the
largest chunck of contiguous memory available?
In regards to 1: I do have some fall back code in place to do that. However,
the buffers are large to improve performance when the data is written to
disk. Writing more small buffers is hurting the application's performance.
In regards to 2: The buffers are large and in memory for performance
reasons. The additional disk access would hurt performance.
In regards to 3: I'm not familiar with AWD. I will take a look at it.
In regards to 4: I tried setting the /LARGEADDRESSAWARE flag when linking,
expecting that to extend the process memory to 3 GB, but it didn't help. Am
I misunderstanding
something? Or, is that only part of it, and setting /3GB in boot.ini is what
makes it work?
Thanks,
Eric
| |
| Severian 2005-03-10, 9:00 pm |
| On Thu, 10 Mar 2005 13:41:05 -0800, Eric Wade
<ewade10@hotmail.com.(donotspam)> wrote:
>
>
>"Severian" wrote:
>
>
>Task Manager indicates that the Mem Usage of the application's process is
>only 1.3 GB. On the surface, this seems to indicate that there should be 700
>MB of unused memory for the application to use. Is there some way to
>programmatically verify this? Is there some way to check the size of the
>largest chunck of contiguous memory available?
You could probably create one using VirtualQueryEx(), but it would be
a mess.
>In regards to 1: I do have some fall back code in place to do that. However,
>the buffers are large to improve performance when the data is written to
>disk. Writing more small buffers is hurting the application's performance.
You might want to look at WriteFileGather().
Also, use VirtualAlloc() for your large allocations to avoid heap
overhead and any possible fragmentation caused by it.
>In regards to 2: The buffers are large and in memory for performance
>reasons. The additional disk access would hurt performance.
Memory mapped file data is not actually written to disk unless
requested or necessary (low system memory, etc.) Think of it as a
"private" page file.
>In regards to 3: I'm not familiar with AWD. I will take a look at it.
>
>In regards to 4: I tried setting the /LARGEADDRESSAWARE flag when linking,
>expecting that to extend the process memory to 3 GB, but it didn't help. Am
>I misunderstanding
>something? Or, is that only part of it, and setting /3GB in boot.ini is what
>makes it work?
Yes, I believe /3GB is required for /LARGEADRESSAWARE to take effect
in the executable, but I've never used it so I'm not sure.
Also, is this some special disk? I can't imagine writing 10 18MB
buffers taking significantly longer than 1 180MB buffer.
Are you using the appropriate flags when you open the file?
(FILE_FLAG_...) You may find that using one or more of these improves
performance.
--
Sev
| |
| Ivan Brugiolo [MSFT] 2005-03-11, 4:07 am |
| > Someone may pop in here and offer the name of a utility which can
> dynamically display your process' memory layout for you.
the `!address` debugger extension command in cdb/ntsd/windbg
displays the layout of the address space of your process,
inluding the largest contiguous availalble region.
Heap fragmentation can be inferred by `!heap -s`.
--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"William DePalo [MVP VC++]" <willd.no.spam@mvps.org> wrote in message
news:OBtIsQbJFHA.2764@tk2msftngp13.phx.gbl...
> "Eric Wade" <ewade10@hotmail.com.(donotspam)> wrote in message
> news:1B8532C7-A30F-497F-ADDC-6E39C3685742@microsoft.com...
>
> Hmm, we are talking about virtual memory here, right? The amount of
physical
> memory that you have may de dependent on what all else is ongoing on that
> box, but if all the DLLs you load have not been rebuilt between iterations
> and if the heap gets allocated at the same spot in virtual memory then you
> should see some consistency.
>
>
> That data point fits perfectly with the virtual memory fragmentation
> hypothesis.
>
> Someone may pop in here and offer the name of a utility which can
> dynamically display your process' memory layout for you. If no one does,
you
> might want to check the docs for EnumProcessModules() and
> GetMoudleInformation(). They should allow you to get information on DLL
> memory usage. And get ProcessHeap() and HeapWalk() should get you some
> information on the memory occupied by your heaps. The information they
> provide should be able to confirm or dismiss the virtual memory
> fragmentation hypothesis.
>
> Regards,
> Will
>
>
| |
| Alexander Grigoriev 2005-03-11, 4:07 am |
| Note that a single I/O operation in Windows cannot exceed 64 MB... (unless
it's very suboptimal cached I/O, causing severe cache bloat).
"Severian" <severian@chlamydia-is-not-a-flower.com> wrote in message
news:1ih1315e9q9m1qtbavrp4c8qagnu17n68m@
4ax.com...
> On Thu, 10 Mar 2005 13:41:05 -0800, Eric Wade
> <ewade10@hotmail.com.(donotspam)> wrote:
>
>
> Also, is this some special disk? I can't imagine writing 10 18MB
> buffers taking significantly longer than 1 180MB buffer.
>
> Are you using the appropriate flags when you open the file?
> (FILE_FLAG_...) You may find that using one or more of these improves
> performance.
>
> --
> Sev
| |
| adebaene@club-internet.fr 2005-03-11, 8:59 am |
|
Eric Wade (donotspam) wrote:
> Task Manager indicates that the Mem Usage of the application's
process is
> only 1.3 GB. On the surface, this seems to indicate that there should
be 700
> MB of unused memory for the application to use. Is there some way to
> programmatically verify this?
On the 2GB, you must have not only allocated memory, but also your code
(EXE and all maped DLLs), your stack(s), etc...
> Is there some way to check the size of the
> largest chunck of contiguous memory available?
Debug your program with WinDbg/ctsd/ntsd and use !vadump command to
display the layout of your process address space. The command line tool
vadump.exe
(http://www.microsoft.com/windows200...ng/vadump-o.asp)
do the same thing.
> In regards to 2: The buffers are large and in memory for performance
> reasons. The additional disk access would hurt performance.
Memory mapped file are not written to disk unless necessary. Btw, it is
the fastest way to do file I/O in Windows (since all others file access
use them under the cover).
>
> In regards to 4: I tried setting the /LARGEADDRESSAWARE flag when
linking,
> expecting that to extend the process memory to 3 GB, but it didn't
help. Am
> I misunderstanding
> something? Or, is that only part of it, and setting /3GB in boot.ini
is what
> makes it work?
You need the /3GB flag in boot.ini, but it is not accessible to all
versions of the OS.
Arnaud
MVP - VC
| |
| Jerry Coffin 2005-03-13, 3:58 am |
| In article <F4296B31-FE44-4896-9D76-C444849BD64B@microsoft.com>,
ewade10@hotmail.com. says...
[ ... ]
> Task Manager indicates that the Mem Usage of the application's process is
> only 1.3 GB. On the surface, this seems to indicate that there should be 700
> MB of unused memory for the application to use. Is there some way to
> programmatically verify this? Is there some way to check the size of the
> largest chunck of contiguous memory available?
Here's a program that uses VirtualQueryEx to walk through a process'
memory space and tells you about what its memory map looks like. One
of these days I should probably get it to produce the output as
pretty graphics, but I'm afraid right now you'll need to look through
bunches of text to find what you're looking for -- though given what
you want, you should be able to pipe the output to grep and then sort
to find the free blocks and then sort them into order by size to find
the biggest free block(s) quickly and easily. Depending on how
capable of a "sort" you have handy, it might be easiest to swap the
positions of the size and address (the first printf in show_modules)
so the size comes first where even the simplest sort program can sort
based on it.
#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <stdio.h>
#include <stdlib.h>
void show_modules(HANDLE process) {
unsigned char *p = NULL;
MEMORY_BASIC_INFORMATION info;
for ( p = NULL;
VirtualQueryEx(process, p, &info, sizeof(info))
== sizeof(info);
p += info.RegionSize )
{
printf("%#10.10x (%6uK)\t",
info.BaseAddress,
info.RegionSize/1024);
switch (info.State) {
case MEM_COMMIT:
printf("Committed");
break;
case MEM_RESERVE:
printf("Reserved");
break;
case MEM_FREE:
printf("Free");
break;
}
printf("\t");
switch (info.Type) {
case MEM_IMAGE:
printf("Code Module");
break;
case MEM_MAPPED:
printf("Mapped ");
break;
case MEM_PRIVATE:
printf("Private ");
}
printf("\t");
int guard = 0, nocache = 0;
if ( info.AllocationProtect & PAGE_NOCACHE)
nocache = 1;
if ( info.AllocationProtect & PAGE_GUARD )
guard = 1;
info.AllocationProtect &= ~(PAGE_GUARD | PAGE_NOCACHE);
switch (info.AllocationProtect) {
case PAGE_READONLY:
printf("Read Only");
break;
case PAGE_READWRITE:
printf("Read/Write");
break;
case PAGE_WRITECOPY:
printf("Copy on Write");
break;
case PAGE_EXECUTE:
printf("Execute only");
break;
case PAGE_EXECUTE_READ:
printf("Execute/Read");
break;
case PAGE_EXECUTE_READWRITE:
printf("Execute/Read/Write");
break;
case PAGE_EXECUTE_WRITECOPY:
printf("COW Executable");
break;
}
if (guard)
printf("\tguard page");
if (nocache)
printf("\tnon-cachable");
printf("\n");
}
}
int main(int argc, char **argv) {
if (argc != 2) {
fprintf(stderr, "Usage: %s <process ID>", argv[0]);
return 1;
}
HANDLE process = OpenProcess(
PROCESS_VM_READ | PROCESS_QUERY_INFORMATION,
false,
atoi(argv[1]));
show_modules(process);
return 0;
}
--
Later,
Jerry.
The universe is a figment of its own imagination.
|
|
|
|
|