loss of data on pipes

From: Paul Pluzhnikov (ppluzhnikov-nsp@charter.net)
Subject: Re: loss of data on pipes

View this article only
Newsgroups: comp.unix.programmer
Date: 2004-12-16 21:51:53 PST

"Adrian  Wong" <atwong@gmail.com> writes:

> Does this sound right?

>No. It sounds like a bug in your program.



> I didnt think pipes would be "lossy"?

>They aren't.

They aren't lossy -if you mean that they should be writing out all
that they *manage* to read in.

> Shouldnt the child processes printing to
> a pipe be blocked if the pipe is full, i.e. the
> whole setup has it's own throttling mechanism?

>Yes.

The best thing to do is to write() in a loop, till all of the bytes
are sent out. I mean, write() returns the no. of bytes actually
written -which you use to determine how much more needs to be written.

> BTW I am interested in any other solution for
> the mangled output problem besides writing to
> regular files.

>The trivial solution is to use write(2) instead of printf(3) in
>the child output. So long as the buffer you write() is smaller
>then PIPE_BUF (32768 on AIX-5.1), your writes will be atomic and no
>"mangling" will happen.

Some unix'es have an upper limit of 4KB. If you are communicating
small amounts of data between related processes -pipes are indeed the
preferred means. For larger amounts, try shared memory or mmap()'ing a
file wherein the child writes to it, and the parent reads in and
deletes it. Both of these mechanisms will require a synchronization
primitive to coordinate between child & parent.
You may want to post some pseudo code for people to point out the
mistake.

regards
-kamal