Using named pipes in father-child IPC

Hello everyone,

I have a C program (the driver) which repeatedly calls a shell script.
The driver and shell script communicate both ways: the driver gives the
script several parameters, and the script returns a string to the driver.

First, I used plain files. But it seems the OS becomes quite busy just
creating and deleting the files, and I don't really need a trace on disk
of the communication. So I thought named pipes might be a better solution?

I don't quite see the sequence of calls to make everything work though.

If I understand correctly, named pipes should only work one way? I
shouldn't use the same pipe to pass parameters TO the script and then
read the result back?

I'll assume I have two named pipes: to_script and from_script.

My first version of the driver was:

1. open to_script for writing.
2. write the parameters.
3. call system("./the_script.sh")

But either 1. or 2. block until someone opens the read end of the named
pipe. Sounds like a deadlock situation...

Does thei mean I can't use system() and that I'll have to work some
fork/exec/wait magic myself if I want to use named pipes?

Or is there some way to use non-blocking named pipes?

--
Regards, Grumble

Grumble wrote:
> Hello everyone,
>
> I have a C program (the driver) which repeatedly calls a shell script.
> The driver and shell script communicate both ways: the driver gives the
> script several parameters, and the script returns a string to the driver.
>
> First, I used plain files. But it seems the OS becomes quite busy just
> creating and deleting the files, and I don't really need a trace on disk
> of the communication. So I thought named pipes might be a better solution?
>
> I don't quite see the sequence of calls to make everything work though.
>
> If I understand correctly, named pipes should only work one way? I
> shouldn't use the same pipe to pass parameters TO the script and then
> read the result back?
>
> I'll assume I have two named pipes: to_script and from_script.
>
> My first version of the driver was:
>
> 1. open to_script for writing.
> 2. write the parameters.
> 3. call system("./the_script.sh")
>
> But either 1. or 2. block until someone opens the read end of the named
> pipe. Sounds like a deadlock situation...
>
> Does thei mean I can't use system() and that I'll have to work some
> fork/exec/wait magic myself if I want to use named pipes?
>
> Or is there some way to use non-blocking named pipes?
>

Hi Grumble,

If you don't want to block when opening pipes, open them with
O_NDELAY (or was it O_NONBLOCK) set ... (open() not fopen())

But why use named pipes at all ??? Why don't you connect your
C-program (driver) to stdin and stdout of the script ??

That's done with
- pipe() // 2 new fd's for unnamed pipe1 (read and write end)
- pipe() // 2 new fd's for unnamed pipe2 (read and write end)
- fork()
- in parent:
- close read-end of pipe1
- close write-end of pipe2
- write to pipe1 write-side (to be read on stdin in child)
- read from pipe2 read-side (what was written to stdout from
script)
- in child:
- close write-end of pipe1
- close read-end of pipe2
- dup2() pipe1Rd to 0 (stdin)
- dup2() pipe2Wr to 1 (stdout)
- exec() script.

Regards ... Rainer

Grumble <devnull@kma.eu.org> writes:

> Hello everyone,
>
> I have a C program (the driver) which repeatedly calls a shell
> script. The driver and shell script communicate both ways: the driver
> gives the script several parameters, and the script returns a string
> to the driver.
>
> First, I used plain files. But it seems the OS becomes quite busy just
> creating and deleting the files, and I don't really need a trace on
> disk of the communication. So I thought named pipes might be a better
> solution?

You don't need named pipes either, just plain pipes.

man 2 pipe

> I don't quite see the sequence of calls to make everything work though.
>
> If I understand correctly, named pipes should only work one way? I
> shouldn't use the same pipe to pass parameters TO the script and then
> read the result back?
>
> I'll assume I have two named pipes: to_script and from_script.
>
> My first version of the driver was:
>
> 1. open to_script for writing.
> 2. write the parameters.
> 3. call system("./the_script.sh")
>
> But either 1. or 2. block until someone opens the read end of the
> named pipe. Sounds like a deadlock situation...
>
> Does thei mean I can't use system() and that I'll have to work some
> fork/exec/wait magic myself if I want to use named pipes?

Yes, it's always better to call fork/exec yourself than to use system.

/* pseudo code follows:
man pipe
man fork
man dup2
man execl
man fread
*/
int c2p[2];
int p2c[2];
int res;
res=pipe(c2p);
res=pipe(p2c);
char result[1000];
switch(child=fork()){
case 0: /* in child */
close(p2c[0]);
close(c2p[1]);
close(0);
close(1);
dup2(p2c[1],0);
dup2(c2p[0],1);
execl("/the_script.sh",0)
case -1: /* error */
exit(1);
default: /* in parent */
close(c2p[1]);
close(p2c[0]);
fprintf(p2c[1],"Data for the script");
fread(result,sizeof(result),1,c2p[0]);
}

And the_script.sh reads and writes its stdin and stdout:

#!/bin/bash
function process () {
..
}
read arguments
result="$(process "$arguments")"
echo "$result"
exit 0

> Or is there some way to use non-blocking named pipes?

No.

--
__Pascal Bourguignon__                     http://www.informatimago.com/

In a World without Walls and Fences,
who needs Windows and Gates?

Rainer Temme wrote:

> Grumble wrote:
> 
>
> Hi Grumble,
>
> If you don't want to block when opening pipes, open them with
>    O_NDELAY (or was it O_NONBLOCK) set ... (open() not fopen())

AFAICT, on my platform (Linux)
open("named_pipe", O_NONBLOCK | O_WRONLY); fails with ENXIO.

The man page for open states:
O_NONBLOCK | O_WRONLY is set, the named file is a  FIFO  and  no
process has the file open for reading.  Or, the file is a device
special file and no corresponding device exists.

Is this not the expected behavior? (I have an aging kernel.)

> But why use named pipes at all ??? Why don't you connect your
>    C-program (driver) to stdin and stdout of the script ??

The script calls several programs that write to stdout and stderr (each
stream might have been redirected to a file). I don't want the script to
pollute the channel used to communicate back to the driver.

For example, assume I run driver 1>stdout.txt 2>stderr.txt

The driver itself does not output anything, but AFAIU, the driver's
children (the script in this case) inherit the redirections for stdout
and stderr. Thus all the programs called by the script write to the two
files. However, the script still needs to report a string back to the
driver, and I don't think it can use stdout or stderr, or am I wrong?

> That's done with
>    - pipe() // 2 new fd's for unnamed pipe1 (read and write end)
>    - pipe() // 2 new fd's for unnamed pipe2 (read and write end)
>    - fork()
>    - in parent:
>       - close read-end of pipe1
>       - close write-end of pipe2
>       - write to pipe1 write-side (to be read on stdin in child)
>       - read from pipe2 read-side (what was written to stdout from
>         script)
>    - in child:
>       - close write-end of pipe1
>       - close read-end of pipe2
>       - dup2() pipe1Rd to 0 (stdin)
>       - dup2() pipe2Wr to 1 (stdout)
>       - exec() script.

Thanks!

--
Regards, Grumble

Grumble wrote:
> AFAICT, on my platform (Linux)
>   open("named_pipe", O_NONBLOCK | O_WRONLY); fails with ENXIO.
>
> The man page for open states:
>   O_NONBLOCK | O_WRONLY is set, the named file is a  FIFO  and  no
>   process has the file open for reading.  Or, the file is a device
>   special file and no corresponding device exists.
>
> Is this not the expected behavior? (I have an aging kernel.)
It probably is the expected behaviour ...
you have to make sure that the reader-sides are opened first, _and_
that the reader-sides are opened with N_DELAY (to prevent from
blocking).
 
>
>
> The script calls several programs that write to stdout and stderr (each
> stream might have been redirected to a file). I don't want the script to
> pollute the channel used to communicate back to the driver.
>
> For example, assume I run driver 1>stdout.txt 2>stderr.txt
>
> The driver itself does not output anything, but AFAIU, the driver's
> children (the script in this case) inherit the redirections for stdout
> and stderr. Thus all the programs called by the script write to the two
> files. However, the script still needs to report a string back to the
> driver, and I don't think it can use stdout or stderr, or am I wrong?
You probably missunderstood what I wrote ...
You are _not_ using the driver-programs stdin/stdout...
You are connecting the scripts stdin/stdout to two filedeskriptors
the driver can write-to and read-from.
Therefore $ driver 1>somewhere.out 2>somewhere.err 0>somewhere.in
is still working fine.

Regards ... Rainer

Rainer Temme wrote:

> Grumble wrote:
> 
[snips]
>
> Hi Grumble,
>
> If you don't want to block when opening pipes, open them with
>    O_NDELAY (or was it O_NONBLOCK) set ... (open() not fopen())
>
> But why use named pipes at all ??? Why don't you connect your
>    C-program (driver) to stdin and stdout of the script ??
>
> That's done with
>    - pipe() // 2 new fd's for unnamed pipe1 (read and write end)
>    - pipe() // 2 new fd's for unnamed pipe2 (read and write end)
>    - fork()
>    - in parent:
>       - close read-end of pipe1
>       - close write-end of pipe2

If the parent will be forking off additional children,
it should make sure those children don't inherit the
pipe FDs to this child.  The additional handles would
keep the parent's closing of the pipe1 write-side
from causing an EOF on the child's stdin.  Setting
close-on-exec on the parent ends after the fork should
do the trick.  (Or arrange for subsequent children to
explicitly close the parent ends of sibling's pipes.)

>       - write to pipe1 write-side (to be read on stdin in child)
>       - read from pipe2 read-side (what was written to stdout from
>         script)
>    - in child:
>       - close write-end of pipe1
>       - close read-end of pipe2
>       - dup2() pipe1Rd to 0 (stdin)
>       - dup2() pipe2Wr to 1 (stdout)

- close pipe1Rd
- close pipe2Wr

If you don't close these extra handles, the parent
will not see child's stdin/out closes (unless the
child completely exits and everything closes.)

>       - exec() script.


--
Clem
"If you push something hard enough, it will fall over."
- Fudd's first law of opposition

Grumble <devnull@kma.eu.org> writes:

> Hello everyone,
>
> I have a C program (the driver) which repeatedly calls a shell script.
> The driver and shell script communicate both ways: the driver gives the
> script several parameters, and the script returns a string to the driver.
>
> First, I used plain files. But it seems the OS becomes quite busy just
> creating and deleting the files, and I don't really need a trace on disk
> of the communication. So I thought named pipes might be a better solution?
>
> I don't quite see the sequence of calls to make everything work though.
>
> If I understand correctly, named pipes should only work one way? I
> shouldn't use the same pipe to pass parameters TO the script and then
> read the result back?
>
> I'll assume I have two named pipes: to_script and from_script.
>
> My first version of the driver was:
>
> 1. open to_script for writing.
> 2. write the parameters.
> 3. call system("./the_script.sh")
>
> But either 1. or 2. block until someone opens the read end of the named
> pipe. Sounds like a deadlock situation...
>
> Does thei mean I can't use system() and that I'll have to work some
> fork/exec/wait magic myself if I want to use named pipes?
>
> Or is there some way to use non-blocking named pipes?

Check out popen() - it encapsulates all of the fork/exec/wait and
pipe manipulation magic under a very convenient API. I sounds like
it might be a good solution to your problem.

-SEan

Sean Burke wrote:

> Grumble <devnull@kma.eu.org> writes:
>
> 
[snip]
>
>
> Check out popen() - it encapsulates all of the fork/exec/wait and
> pipe manipulation magic under a very convenient API. I sounds like
> it might be a good solution to your problem.
>
> -SEan

Popen() will not work because the OP says the communications are
bidirectional.

--
Clem
"If you push something hard enough, it will fall over."
- Fudd's first law of opposition

"Mr. Uh Clem" <uhclem@DutchElmSt.invalid> writes:

> Sean Burke wrote:
> 
> [snip] 
>
> Popen() will not work because the OP says the communications are
> bidirectional.

I'm not sure, but it seems like the "several parameters" could as easily
be passed via the arg list, rather than stdin. Also, on some Unix the pipe
is bidirectional. The code below works on Solaris and BSD, but not Linux,
which has uni-directional pipes.

-SEan


#include <stdio.h>
#include <unistd.h>

main()
{
int    fd;
FILE * fp = popen("telnet localhost 22", "w");
if (pipe < 0) {
perror("pipe");
exit(1);
}
fd = fileno(fp);

while (1)
{
fd_set	fds;
int	numfds = fd + 1;

/* Add fd's for stdin, pipe.
*/
FD_ZERO(&fds);
FD_SET(0, &fds);
FD_SET(fd, &fds);

if (select(numfds, &fds, NULL, NULL, NULL) > 0)
{
if (FD_ISSET(0, &fds))
{
/* Read data from stdin and send it to telnet.
*/
char buff[128];
int  n =  read(0, buff, sizeof(buff));

if (n > 0)
write(fd, buff, n);
else
break;
}

if (FD_ISSET(fd, &fds))
{
/* Read data from pipe and send it to stdout.
*/
char buff[128];
int  n =  read(fd, buff, sizeof(buff));

if (n > 0)
write(1, buff, n);
else
break;
}
}
}

exit(0);
}

Mr. Uh Clem wrote:
> Rainer Temme wrote: 

>         If the parent will be forking off additional children,
>         it should make sure those children don't inherit the
>         pipe FDs to this child.  The additional handles would
>         keep the parent's closing of the pipe1 write-side
>         from causing an EOF on the child's stdin.  Setting
>         close-on-exec on the parent ends after the fork should
>         do the trick.  (Or arrange for subsequent children to
>         explicitly close the parent ends of sibling's pipes.)
Yes, one should prevent these fd's from being inherited to
other children forked off later. (I should have thought about
this, since this is an error that caused a colleague of mine
to spend a w behind a debugger).
BUT ... close-on-exec is not the cure ... as the name says it
closes on exec()'s ... but if you only fork(), and then stay
in the forked process (which is not an unlikely scenario)
close-on-exec doesn't do anything!!!
This is obviously a point to keep in mind when trying to handle
multiple concurrent children ... because it cannot be cured right
here, right now.
 

>         - close pipe1Rd
>         - close pipe2Wr
>           If you don't close these extra handles, the parent
>           will not see child's stdin/out closes (unless the
>           child completely exits and everything closes.)

Yes, true again ... so far (whenever I used this construction) I kept
fd's open util exit()/_exit() was called ... because usually
I want to transfer some kind of final information about the execution
done ...
but generally you're right ... when you don't need the communication
any more (and you want to let the parent know that no more data is to
come) call close() ... even if you have to do some more computaion
in the child ... this way the parent knows, that it doesn't need to
wait for data any more and can proceed. (Don't forget to call wait()
or the family-buddies of wait() later).

Rainer