Home > Archive > Unix Programming > June 2007 > Largest memory address accessible by a 32-bit process
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Largest memory address accessible by a 32-bit process
|
|
| scorpneo@gmail.com 2007-06-19, 7:09 pm |
| Hi,
My application needs to figure out the total amount of addressable
memory that is available to it when running and automatically prevent
memory allocations when such a _hard_ memory limit is reached. For 32-
bit machines, I understand that the theoretical limit is 4GB. However,
depending on the memory architecture of the platform, using 4GB as a
limit is not guaranteed to work. For instance, on Win32 user address
space is limited to 2GB (3GB with /3GB switch).
On UNIX, it again seems that different flavors have different memory
architectures and therefore, in theory, different _hard_ memory
limits. For instance on HP-UX, one might be able to use memory in all
4 quadrants if the exe is linked in so and so fashion etc. What I am
looking for is a generic solution where I can at run time query the
platform for the largest available address that is accessible by my
process and then use that to decide the _hard_ memory limit.
I'd appreciate if any of you can point me to a method / API that would
provide this informaiton. A generic solution is preferred, but plat
specific ones are welcome too. On Windows, GetSystemInfo() provides a
struct that has the lpMaximumApplicationAddress value which I use to
figure out this limit.
Thanks :)
| |
| Eric Sosman 2007-06-19, 7:09 pm |
| scorpneo@gmail.com wrote On 06/19/07 17:56,:
> Hi,
>
> My application needs to figure out the total amount of addressable
> memory that is available to it when running and automatically prevent
> memory allocations when such a _hard_ memory limit is reached.
Why?
Seriously, why? The system will enforce whatever
limits exist: you will get NULL from malloc(), or -1
from sbrk(), or some other kind of failure indication
from whatever other API you use to obtain memory. There
is no need to police yourself; the system will do all
the policing that's needed.
If your program discovers what the hard limit is,
there is still no guarantee that you will actually be
able to allocate that much memory. There may be "soft"
limits as well, so your program must in any case be
prepared to cope with allocation failures. And even if
you succeed in allocating memory, there's no guarantee
that successful allocation is a Good Thing for the system
as a whole; see the recent thread "Available Physical
Memory" for a discussion of some of the pitfalls.
So again: Why do you think your program needs to
discover the hard limit before attempting allocations?
--
Eric.Sosman@sun.com
| |
| scorpneo@gmail.com 2007-06-19, 7:09 pm |
| Okay, this is only a part of the bigger picture. The ultimate intent
is to detect situations where the application is operating close to
hard/soft/user-specified limits and proactively unload "aged" data
such that it can honor requests to load new data. Now, in that sense,
if my 32-bit process is able to see all 4GB of RAM on a LINUX box,
then maybe I can wait until I reach about 3900 MB before I start my
unloader, as opposed to on a Win32 where I'd do the unloading at ~1800
MB.
On Jun 19, 5:17 pm, Eric Sosman <Eric.Sos...@sun.com> wrote:
> scorp...@gmail.com wrote On 06/19/07 17:56,:
>
>
>
> Why?
>
> Seriously, why? The system will enforce whatever
> limits exist: you will get NULL from malloc(), or -1
> from sbrk(), or some other kind of failure indication
> from whatever other API you use to obtain memory. There
> is no need to police yourself; the system will do all
> the policing that's needed.
>
> If your program discovers what the hard limit is,
> there is still no guarantee that you will actually be
> able to allocate that much memory. There may be "soft"
> limits as well, so your program must in any case be
> prepared to cope with allocation failures. And even if
> you succeed in allocating memory, there's no guarantee
> that successful allocation is a Good Thing for the system
> as a whole; see the recent thread "Available Physical
> Memory" for a discussion of some of the pitfalls.
>
> So again: Why do you think your program needs to
> discover the hard limit before attempting allocations?
>
> --
> Eric.Sos...@sun.com
| |
| Casper H.S. Dik 2007-06-19, 7:09 pm |
| scorpneo@gmail.com writes:
>Okay, this is only a part of the bigger picture. The ultimate intent
>is to detect situations where the application is operating close to
>hard/soft/user-specified limits and proactively unload "aged" data
>such that it can honor requests to load new data. Now, in that sense,
>if my 32-bit process is able to see all 4GB of RAM on a LINUX box,
>then maybe I can wait until I reach about 3900 MB before I start my
>unloader, as opposed to on a Win32 where I'd do the unloading at ~1800
>MB.
The only reliable way to do so is using a memory allocator which
can call a cleanup routine when it runs low.
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
| |
| Michal Nazarewicz 2007-06-19, 7:09 pm |
| >> scorp...@gmail.com wrote On 06/19/07 17:56,:
scorpneo@gmail.com writes:[color=darkred]
> Okay, this is only a part of the bigger picture. The ultimate intent
> is to detect situations where the application is operating close to
> hard/soft/user-specified limits and proactively unload "aged" data
> such that it can honor requests to load new data. Now, in that sense,
> if my 32-bit process is able to see all 4GB of RAM on a LINUX box,
> then maybe I can wait until I reach about 3900 MB before I start my
> unloader, as opposed to on a Win32 where I'd do the unloading at ~1800
> MB.
You start your unloader when malloc() returns null.
--
Best regards, _ _
.o. | Liege of Serenly Enlightened Majesty of o' \,=./ `o
..o | Computer Science, Michal "mina86" Nazarewicz (o o)
ooo +--<mina86*tlen.pl>---<jid:mina86*chrome.pl>--ooO--(_)--Ooo--
| |
| Chris Friesen 2007-06-19, 10:07 pm |
| Eric Sosman wrote:
> Seriously, why? The system will enforce whatever
> limits exist: you will get NULL from malloc(), or -1
> from sbrk(), or some other kind of failure indication
> from whatever other API you use to obtain memory.
Note that with most linux systems unless you ask for a ridiculously
large number the memory allocation will succeed. The memory will only
be actually allocated when you attempt to access it, so the first
indication of failure is likely to be a SIGSEGV. You'll need to be able
to handle that (likely via siglongjmp()) and then start shedding stale
data to free up memory.
One way around this is to wrap your allocator with some code that
immediately touches every page of memory that was allocated (forcing it
to fault in), and possibly also calling
mlockall(MCL_CURRENT|MCL_FUTURE). This will mean that your app cannot
swap out, but will always be in memory. Note that many systems have
limits on how much memory can be locked by a process.
Chris
| |
| Eric Sosman 2007-06-19, 10:07 pm |
| scorpneo@gmail.com wrote:
[Please don't top-post; I've fixed it in this reply]
> On Jun 19, 5:17 pm, Eric Sosman <Eric.Sos...@sun.com> wrote:
>
> Okay, this is only a part of the bigger picture. The ultimate intent
> is to detect situations where the application is operating close to
> hard/soft/user-specified limits and proactively unload "aged" data
> such that it can honor requests to load new data. Now, in that sense,
> if my 32-bit process is able to see all 4GB of RAM on a LINUX box,
> then maybe I can wait until I reach about 3900 MB before I start my
> unloader, as opposed to on a Win32 where I'd do the unloading at ~1800
> MB.
Once again, I'll suggest that you read the "Available
Physical Memory" thread. And once again, I'll assert that
the hard limits are probably not the important concerns.
--
Eric Sosman
esosman@acm-dot-org.invalid
| |
| Casper H.S. Dik 2007-06-20, 8:06 am |
| Chris Friesen <cbf123@mail.usask.ca> writes:
>Eric Sosman wrote:
[color=darkred]
>Note that with most linux systems unless you ask for a ridiculously
>large number the memory allocation will succeed. The memory will only
>be actually allocated when you attempt to access it, so the first
>indication of failure is likely to be a SIGSEGV. You'll need to be able
>to handle that (likely via siglongjmp()) and then start shedding stale
>data to free up memory.
Except that that cannot work: you've caught a SIGSEGV and you will actually
need to unmap pages and not just free the memory; there's no telling that
if you succeed in doing that the next time you touch the denied page it
will then be allocated.
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
| |
| Maxim Yegorushkin 2007-06-20, 8:06 am |
| On 19 Jun, 23:28, scorp...@gmail.com wrote:
> Okay, this is only a part of the bigger picture. The ultimate intent
> is to detect situations where the application is operating close to
> hard/soft/user-specified limits and proactively unload "aged" data
> such that it can honor requests to load new data. Now, in that sense,
> if my 32-bit process is able to see all 4GB of RAM on a LINUX box,
> then maybe I can wait until I reach about 3900 MB before I start my
> unloader, as opposed to on a Win32 where I'd do the unloading at ~1800
> MB.
You could allocate and commit your memory using mmap() (on Linux you
might commit the memory using MAP_POPULATE flag, as well as MAP_LOCKED
if locking is desirable). Once allocation or committing memory fails,
free some memory and continue.
Does it make sense for your task?
| |
| scorpneo@gmail.com 2007-06-21, 10:05 pm |
| On Jun 20, 2:58 am, Maxim Yegorushkin <maxim.yegorush...@gmail.com>
wrote:
> On 19 Jun, 23:28, scorp...@gmail.com wrote:
>
>
> You could allocate and commit your memory using mmap() (on Linux you
> might commit the memory using MAP_POPULATE flag, as well as MAP_LOCKED
> if locking is desirable). Once allocation or committing memory fails,
> free some memory and continue.
>
> Does it make sense for your task?
The way my unloader works is that it constantly "attempts" to prevent
the application from reaching a state where "a memory allocation
fails". It is run periodically, where it wakes up every once in a
while, checks the memory stats and does its thing. Now, for all of
this to work, the key is to be able to reliably supply the memory
stats. On Win32, I can say, 2GB is the absolute address limit for any
user space process(for now lets say this is the hard limit), and I
know that ~1700 MB (soft limit) Win32 starts acting flaky, so I will
let my unloader get aggressive when the total memory committed for my
process is about 1700 MB. All I need is to be able to trigger the
unloader similarly across most UNIXes. Assuming that most of the
allocator/memory tracking/unloading part of the work is solved, what
are my options to figure out the address limits of a 32-bit process?
For this discussion, we can simply assume that 4GB of memory hardware
is installed on all.
| |
| Spoon 2007-06-21, 10:05 pm |
| scorpneo wrote:
> The way my unloader works is that it constantly "attempts" to prevent
> the application from reaching a state where "a memory allocation
> fails". It is run periodically, where it wakes up every once in a
> while, checks the memory stats and does its thing. Now, for all of
> this to work, the key is to be able to reliably supply the memory
> stats. On Win32, I can say, 2GB is the absolute address limit for any
> user space process(for now lets say this is the hard limit), and I
> know that ~1700 MB (soft limit) Win32 starts acting flaky, so I will
> let my unloader get aggressive when the total memory committed for my
> process is about 1700 MB.
On a system with e.g. 256 MB of RAM and plenty swap space, things just
might get ugly long before you reach 1.7 GB.
| |
| Spoon 2007-06-21, 10:05 pm |
| Chris Friesen wrote:
> Eric Sosman wrote:
>
>
> Note that with most linux systems unless you ask for a ridiculously
> large number the memory allocation will succeed. The memory will only
> be actually allocated when you attempt to access it, so the first
> indication of failure is likely to be a SIGSEGV. You'll need to be able
> to handle that (likely via siglongjmp()) and then start shedding stale
> data to free up memory.
The sy min might also consider disabling overcommit.
|
|
|
|
|