Home > Archive > Unix Programming > September 2006 > Solaris fork() to generate core file
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Solaris fork() to generate core file
|
|
|
| I'm using Intel Solaris 9. I want to get a core dump when my process
hits an error condition, but keep the process running so I do it like
this...
Fork the process using fork(). Parent just carries on running, child
calls abort() to dump core file then exit. Because fork() uses
copy-on-write, one of two things happens:
(i) If the parent modifies very little of its memory while the core is
being dumped, very little extra memory is required.
(ii) If the parent modifies lots of its memory while the core is being
dumped, copies of all the parent's modified pages are made in the
child, requiring lots of extra memory.
This all works fine, until my process gets to a certain size. Once my
process is using over half the available RAM _and_ the parent is
heavily modifying its memory while the core is being dumped, then I run
out of RAM because the child ends up copying all of the parent's
memory.
So, how do I get this to work?
- I can't use swap memory because it kills performance.
- I'd prefer not to install more RAM. Cost is prohibitive.
- Is there a different way to get a core file? The parent process
_must_ keep running and not be impacted or delayed by dumping core.
- Is there any way to get the child to free off memory as it dumps
core, rather than waiting until it has dumped the entire core? This is
all inside abort() so appears outside of my control.
Thanks for any ideas.
| |
| Nils O. Selåsdal 2006-09-14, 7:01 pm |
| NickB wrote:
> I'm using Intel Solaris 9. I want to get a core dump when my process
> hits an error condition, but keep the process running so I do it like
> this...
>
> Fork the process using fork(). Parent just carries on running, child
> calls abort() to dump core file then exit. Because fork() uses
> copy-on-write, one of two things happens:
> (i) If the parent modifies very little of its memory while the core is
> being dumped, very little extra memory is required.
> (ii) If the parent modifies lots of its memory while the core is being
> dumped, copies of all the parent's modified pages are made in the
> child, requiring lots of extra memory.
>
> This all works fine, until my process gets to a certain size. Once my
> process is using over half the available RAM _and_ the parent is
> heavily modifying its memory while the core is being dumped, then I run
> out of RAM because the child ends up copying all of the parent's
> memory.
Can't the parent simply wait until the child is finished dumping core
before it goes on , thus little memory should be copied since neither
modifies anything ?
| |
|
| Sorry, no. I'm trying to get diagnostics (core file) from a live
production system which can't block processing of incoming data for
more than a second.
I find this method of getting core files from a live system very
successful, except when approaching the limit of available memory as
here. I've got a process about 2GB in size and this takes around two
minutes to dump the core file.
Nils O. Sel=E5sdal wrote:
> Can't the parent simply wait until the child is finished dumping core
> before it goes on , thus little memory should be copied since neither
> modifies anything ?
| |
| Henry Townsend 2006-09-14, 7:01 pm |
| NickB wrote:
> This all works fine, until my process gets to a certain size. Once my
> process is using over half the available RAM _and_ the parent is
> heavily modifying its memory while the core is being dumped, then I run
> out of RAM because the child ends up copying all of the parent's
> memory.
Isn't this what vfork is for?
HT
| |
| Casper H.S. Dik 2006-09-14, 7:01 pm |
| "NickB" <butchernick@hotmail.com> writes:
>This all works fine, until my process gets to a certain size. Once my
>process is using over half the available RAM _and_ the parent is
>heavily modifying its memory while the core is being dumped, then I run
>out of RAM because the child ends up copying all of the parent's
>memory.
In Solaris the fork() will fail because the fork() will try to
reserve all the possibly needed virtual memory.
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
| |
| Barry Margolin 2006-09-14, 10:00 pm |
| In article <1158240415.146956.280560@e3g2000cwe.googlegroups.com>,
"NickB" <butchernick@hotmail.com> wrote:
> I'm using Intel Solaris 9. I want to get a core dump when my process
> hits an error condition, but keep the process running so I do it like
> this...
>
> Fork the process using fork(). Parent just carries on running, child
> calls abort() to dump core file then exit. Because fork() uses
> copy-on-write, one of two things happens:
> (i) If the parent modifies very little of its memory while the core is
> being dumped, very little extra memory is required.
> (ii) If the parent modifies lots of its memory while the core is being
> dumped, copies of all the parent's modified pages are made in the
> child, requiring lots of extra memory.
>
> This all works fine, until my process gets to a certain size. Once my
> process is using over half the available RAM _and_ the parent is
> heavily modifying its memory while the core is being dumped, then I run
> out of RAM because the child ends up copying all of the parent's
> memory.
I doubt there's any way to tune this. Core dumps are not expected to be
a regular occurrence, and OS designers rarely spend much effort
optimizing error cases.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
| |
|
| I have sufficient virtual memory that fork() succeeds. But I want to
avoid the core dumping process actually using virtual memory because
that kills performance.
I'm coming to the conclusion that it simply isn't possible to do this
without adding RAM.
>
> In Solaris the fork() will fail because the fork() will try to
> reserve all the possibly needed virtual memory.
>
| |
| Casper H.S. Dik 2006-09-15, 8:01 am |
| "NickB" <butchernick@hotmail.com> writes:
>I have sufficient virtual memory that fork() succeeds. But I want to
>avoid the core dumping process actually using virtual memory because
>that kills performance.
The child reads and does not write pages on the core dump; so it will *never*
cause new pages to be allocated; if the parent modifies the pages,
there is already a second copy due to the COW fault.
If you're seeing a performance problem it might just as well be caused by
the memory evicted because of the large amount of I/O or the fact that
the child needs to page its copies of the files back in.
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
| |
| Barry Margolin 2006-09-15, 7:00 pm |
| In article <450a9126$0$4521$e4fe514c@news.xs4all.nl>,
Casper H.S. Dik <Casper.Dik@Sun.COM> wrote:
> "NickB" <butchernick@hotmail.com> writes:
>
>
> The child reads and does not write pages on the core dump; so it will *never*
> cause new pages to be allocated; if the parent modifies the pages,
> there is already a second copy due to the COW fault.
>
> If you're seeing a performance problem it might just as well be caused by
> the memory evicted because of the large amount of I/O or the fact that
> the child needs to page its copies of the files back in.
That's his point. He can't handle the performance impact of paging, he
needs all the virtual memory to stay in RAM. He's able to accomplish
this as long as there's only one process, but when there are two
processes and the parent modifies lots of stuff the COW causes his VM to
exceed his RAM, resulting in performance-impacting thrashing.
The basic problem is that he hasn't "overengineered" his system. As a
general rule, if you build a system that just barely meets the resource
requirements of your application, you're going to run into problems when
the requirements increase in the slightest. You need to allow some
breathing room, because requirements almost always change over time.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
| |
| David Schwartz 2006-09-15, 9:59 pm |
|
NickB wrote:
> I have sufficient virtual memory that fork() succeeds. But I want to
> avoid the core dumping process actually using virtual memory because
> that kills performance.
>
> I'm coming to the conclusion that it simply isn't possible to do this
> without adding RAM.
Yeah. If you have a machine that's being run for debug and testing
purposes on live data, it usually needs to have much more CPU and
memory as one that's not being used for debug and testing. Otherwise,
there just isn't enough horsepower for debugging.
We use "live debug" builds on lots of our programs to stress test them
and catch bugs. Our "live debug" builds include dynamic lock order
analysis, tracking of memory allocation by file and line, and the
ability to fork/abort when suspicious events are detected, such as a
lock being held for too long or acquired with a conflicting lock held.
They track locks held by file/line that acquired them, and can dump
that information on demand.
If you want to catch things like double-frees and access after free
errors, you need to oversize many memory allocations and keep track of
the file/line where blocks were allocated and freed. This takes extra
memory and CPU too.
The machines that run these builds generally have twice the CPU and
memory they would have if they didn't do these kinds of extra things.
DS
|
|
|
|
|