Home > Archive > Unix Programming > April 2005 > Using named pipes in father-child IPC
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Using named pipes in father-child IPC
|
|
| Grumble 2005-04-18, 3:58 pm |
| Hello everyone,
I have a C program (the driver) which repeatedly calls a shell script.
The driver and shell script communicate both ways: the driver gives the
script several parameters, and the script returns a string to the driver.
First, I used plain files. But it seems the OS becomes quite busy just
creating and deleting the files, and I don't really need a trace on disk
of the communication. So I thought named pipes might be a better solution?
I don't quite see the sequence of calls to make everything work though.
If I understand correctly, named pipes should only work one way? I
shouldn't use the same pipe to pass parameters TO the script and then
read the result back?
I'll assume I have two named pipes: to_script and from_script.
My first version of the driver was:
1. open to_script for writing.
2. write the parameters.
3. call system("./the_script.sh")
But either 1. or 2. block until someone opens the read end of the named
pipe. Sounds like a deadlock situation...
Does thei mean I can't use system() and that I'll have to work some
fork/exec/wait magic myself if I want to use named pipes?
Or is there some way to use non-blocking named pipes?
--
Regards, Grumble
| |
| Rainer Temme 2005-04-18, 3:58 pm |
| Grumble wrote:
> Hello everyone,
>
> I have a C program (the driver) which repeatedly calls a shell script.
> The driver and shell script communicate both ways: the driver gives the
> script several parameters, and the script returns a string to the driver.
>
> First, I used plain files. But it seems the OS becomes quite busy just
> creating and deleting the files, and I don't really need a trace on disk
> of the communication. So I thought named pipes might be a better solution?
>
> I don't quite see the sequence of calls to make everything work though.
>
> If I understand correctly, named pipes should only work one way? I
> shouldn't use the same pipe to pass parameters TO the script and then
> read the result back?
>
> I'll assume I have two named pipes: to_script and from_script.
>
> My first version of the driver was:
>
> 1. open to_script for writing.
> 2. write the parameters.
> 3. call system("./the_script.sh")
>
> But either 1. or 2. block until someone opens the read end of the named
> pipe. Sounds like a deadlock situation...
>
> Does thei mean I can't use system() and that I'll have to work some
> fork/exec/wait magic myself if I want to use named pipes?
>
> Or is there some way to use non-blocking named pipes?
>
Hi Grumble,
If you don't want to block when opening pipes, open them with
O_NDELAY (or was it O_NONBLOCK) set ... (open() not fopen())
But why use named pipes at all ??? Why don't you connect your
C-program (driver) to stdin and stdout of the script ??
That's done with
- pipe() // 2 new fd's for unnamed pipe1 (read and write end)
- pipe() // 2 new fd's for unnamed pipe2 (read and write end)
- fork()
- in parent:
- close read-end of pipe1
- close write-end of pipe2
- write to pipe1 write-side (to be read on stdin in child)
- read from pipe2 read-side (what was written to stdout from
script)
- in child:
- close write-end of pipe1
- close read-end of pipe2
- dup2() pipe1Rd to 0 (stdin)
- dup2() pipe2Wr to 1 (stdout)
- exec() script.
Regards ... Rainer
| |
| Pascal Bourguignon 2005-04-18, 3:58 pm |
| Grumble <devnull@kma.eu.org> writes:
> Hello everyone,
>
> I have a C program (the driver) which repeatedly calls a shell
> script. The driver and shell script communicate both ways: the driver
> gives the script several parameters, and the script returns a string
> to the driver.
>
> First, I used plain files. But it seems the OS becomes quite busy just
> creating and deleting the files, and I don't really need a trace on
> disk of the communication. So I thought named pipes might be a better
> solution?
You don't need named pipes either, just plain pipes.
man 2 pipe
> I don't quite see the sequence of calls to make everything work though.
>
> If I understand correctly, named pipes should only work one way? I
> shouldn't use the same pipe to pass parameters TO the script and then
> read the result back?
>
> I'll assume I have two named pipes: to_script and from_script.
>
> My first version of the driver was:
>
> 1. open to_script for writing.
> 2. write the parameters.
> 3. call system("./the_script.sh")
>
> But either 1. or 2. block until someone opens the read end of the
> named pipe. Sounds like a deadlock situation...
>
> Does thei mean I can't use system() and that I'll have to work some
> fork/exec/wait magic myself if I want to use named pipes?
Yes, it's always better to call fork/exec yourself than to use system.
/* pseudo code follows:
man pipe
man fork
man dup2
man execl
man fread
*/
int c2p[2];
int p2c[2];
int res;
res=pipe(c2p);
res=pipe(p2c);
char result[1000];
switch(child=fork()){
case 0: /* in child */
close(p2c[0]);
close(c2p[1]);
close(0);
close(1);
dup2(p2c[1],0);
dup2(c2p[0],1);
execl("/the_script.sh",0)
case -1: /* error */
exit(1);
default: /* in parent */
close(c2p[1]);
close(p2c[0]);
fprintf(p2c[1],"Data for the script");
fread(result,sizeof(result),1,c2p[0]);
}
And the_script.sh reads and writes its stdin and stdout:
#!/bin/bash
function process () {
...
}
read arguments
result="$(process "$arguments")"
echo "$result"
exit 0
> Or is there some way to use non-blocking named pipes?
No.
--
__Pascal Bourguignon__ http://www.informatimago.com/
In a World without Walls and Fences,
who needs Windows and Gates?
| |
| Grumble 2005-04-18, 3:58 pm |
| Rainer Temme wrote:
> Grumble wrote:
>
>
> Hi Grumble,
>
> If you don't want to block when opening pipes, open them with
> O_NDELAY (or was it O_NONBLOCK) set ... (open() not fopen())
AFAICT, on my platform (Linux)
open("named_pipe", O_NONBLOCK | O_WRONLY); fails with ENXIO.
The man page for open states:
O_NONBLOCK | O_WRONLY is set, the named file is a FIFO and no
process has the file open for reading. Or, the file is a device
special file and no corresponding device exists.
Is this not the expected behavior? (I have an aging kernel.)
> But why use named pipes at all ??? Why don't you connect your
> C-program (driver) to stdin and stdout of the script ??
The script calls several programs that write to stdout and stderr (each
stream might have been redirected to a file). I don't want the script to
pollute the channel used to communicate back to the driver.
For example, assume I run driver 1>stdout.txt 2>stderr.txt
The driver itself does not output anything, but AFAIU, the driver's
children (the script in this case) inherit the redirections for stdout
and stderr. Thus all the programs called by the script write to the two
files. However, the script still needs to report a string back to the
driver, and I don't think it can use stdout or stderr, or am I wrong?
> That's done with
> - pipe() // 2 new fd's for unnamed pipe1 (read and write end)
> - pipe() // 2 new fd's for unnamed pipe2 (read and write end)
> - fork()
> - in parent:
> - close read-end of pipe1
> - close write-end of pipe2
> - write to pipe1 write-side (to be read on stdin in child)
> - read from pipe2 read-side (what was written to stdout from
> script)
> - in child:
> - close write-end of pipe1
> - close read-end of pipe2
> - dup2() pipe1Rd to 0 (stdin)
> - dup2() pipe2Wr to 1 (stdout)
> - exec() script.
Thanks!
--
Regards, Grumble
| |
| Rainer Temme 2005-04-18, 3:58 pm |
| Grumble wrote:
> AFAICT, on my platform (Linux)
> open("named_pipe", O_NONBLOCK | O_WRONLY); fails with ENXIO.
>
> The man page for open states:
> O_NONBLOCK | O_WRONLY is set, the named file is a FIFO and no
> process has the file open for reading. Or, the file is a device
> special file and no corresponding device exists.
>
> Is this not the expected behavior? (I have an aging kernel.)
It probably is the expected behaviour ...
you have to make sure that the reader-sides are opened first, _and_
that the reader-sides are opened with N_DELAY (to prevent from
blocking).
>
>
> The script calls several programs that write to stdout and stderr (each
> stream might have been redirected to a file). I don't want the script to
> pollute the channel used to communicate back to the driver.
>
> For example, assume I run driver 1>stdout.txt 2>stderr.txt
>
> The driver itself does not output anything, but AFAIU, the driver's
> children (the script in this case) inherit the redirections for stdout
> and stderr. Thus all the programs called by the script write to the two
> files. However, the script still needs to report a string back to the
> driver, and I don't think it can use stdout or stderr, or am I wrong?
You probably missunderstood what I wrote ...
You are _not_ using the driver-programs stdin/stdout...
You are connecting the scripts stdin/stdout to two filedeskriptors
the driver can write-to and read-from.
Therefore $ driver 1>somewhere.out 2>somewhere.err 0>somewhere.in
is still working fine.
Regards ... Rainer
| |
| Mr. Uh Clem 2005-04-18, 3:58 pm |
| Rainer Temme wrote:
> Grumble wrote:
>
[snips][color=darkred]
>
> Hi Grumble,
>
> If you don't want to block when opening pipes, open them with
> O_NDELAY (or was it O_NONBLOCK) set ... (open() not fopen())
>
> But why use named pipes at all ??? Why don't you connect your
> C-program (driver) to stdin and stdout of the script ??
>
> That's done with
> - pipe() // 2 new fd's for unnamed pipe1 (read and write end)
> - pipe() // 2 new fd's for unnamed pipe2 (read and write end)
> - fork()
> - in parent:
> - close read-end of pipe1
> - close write-end of pipe2
If the parent will be forking off additional children,
it should make sure those children don't inherit the
pipe FDs to this child. The additional handles would
keep the parent's closing of the pipe1 write-side
from causing an EOF on the child's stdin. Setting
close-on-exec on the parent ends after the fork should
do the trick. (Or arrange for subsequent children to
explicitly close the parent ends of sibling's pipes.)
> - write to pipe1 write-side (to be read on stdin in child)
> - read from pipe2 read-side (what was written to stdout from
> script)
> - in child:
> - close write-end of pipe1
> - close read-end of pipe2
> - dup2() pipe1Rd to 0 (stdin)
> - dup2() pipe2Wr to 1 (stdout)
- close pipe1Rd
- close pipe2Wr
If you don't close these extra handles, the parent
will not see child's stdin/out closes (unless the
child completely exits and everything closes.)
> - exec() script.
--
Clem
"If you push something hard enough, it will fall over."
- Fudd's first law of opposition
| |
| Sean Burke 2005-04-18, 8:58 pm |
|
Grumble <devnull@kma.eu.org> writes:
> Hello everyone,
>
> I have a C program (the driver) which repeatedly calls a shell script.
> The driver and shell script communicate both ways: the driver gives the
> script several parameters, and the script returns a string to the driver.
>
> First, I used plain files. But it seems the OS becomes quite busy just
> creating and deleting the files, and I don't really need a trace on disk
> of the communication. So I thought named pipes might be a better solution?
>
> I don't quite see the sequence of calls to make everything work though.
>
> If I understand correctly, named pipes should only work one way? I
> shouldn't use the same pipe to pass parameters TO the script and then
> read the result back?
>
> I'll assume I have two named pipes: to_script and from_script.
>
> My first version of the driver was:
>
> 1. open to_script for writing.
> 2. write the parameters.
> 3. call system("./the_script.sh")
>
> But either 1. or 2. block until someone opens the read end of the named
> pipe. Sounds like a deadlock situation...
>
> Does thei mean I can't use system() and that I'll have to work some
> fork/exec/wait magic myself if I want to use named pipes?
>
> Or is there some way to use non-blocking named pipes?
Check out popen() - it encapsulates all of the fork/exec/wait and
pipe manipulation magic under a very convenient API. I sounds like
it might be a good solution to your problem.
-SEan
| |
| Mr. Uh Clem 2005-04-18, 8:58 pm |
| Sean Burke wrote:
> Grumble <devnull@kma.eu.org> writes:
>
>
[snip][color=darkred]
>
>
> Check out popen() - it encapsulates all of the fork/exec/wait and
> pipe manipulation magic under a very convenient API. I sounds like
> it might be a good solution to your problem.
>
> -SEan
Popen() will not work because the OP says the communications are
bidirectional.
--
Clem
"If you push something hard enough, it will fall over."
- Fudd's first law of opposition
| |
| Sean Burke 2005-04-19, 3:59 am |
|
"Mr. Uh Clem" <uhclem@DutchElmSt.invalid> writes:
> Sean Burke wrote:
>
> [snip]
>
> Popen() will not work because the OP says the communications are
> bidirectional.
I'm not sure, but it seems like the "several parameters" could as easily
be passed via the arg list, rather than stdin. Also, on some Unix the pipe
is bidirectional. The code below works on Solaris and BSD, but not Linux,
which has uni-directional pipes.
-SEan
#include <stdio.h>
#include <unistd.h>
main()
{
int fd;
FILE * fp = popen("telnet localhost 22", "w");
if (pipe < 0) {
perror("pipe");
exit(1);
}
fd = fileno(fp);
while (1)
{
fd_set fds;
int numfds = fd + 1;
/* Add fd's for stdin, pipe.
*/
FD_ZERO(&fds);
FD_SET(0, &fds);
FD_SET(fd, &fds);
if (select(numfds, &fds, NULL, NULL, NULL) > 0)
{
if (FD_ISSET(0, &fds))
{
/* Read data from stdin and send it to telnet.
*/
char buff[128];
int n = read(0, buff, sizeof(buff));
if (n > 0)
write(fd, buff, n);
else
break;
}
if (FD_ISSET(fd, &fds))
{
/* Read data from pipe and send it to stdout.
*/
char buff[128];
int n = read(fd, buff, sizeof(buff));
if (n > 0)
write(1, buff, n);
else
break;
}
}
}
exit(0);
}
| |
| Rainer Temme 2005-04-19, 8:57 am |
| Mr. Uh Clem wrote:
> Rainer Temme wrote:
[color=darkred]
> If the parent will be forking off additional children,
> it should make sure those children don't inherit the
> pipe FDs to this child. The additional handles would
> keep the parent's closing of the pipe1 write-side
> from causing an EOF on the child's stdin. Setting
> close-on-exec on the parent ends after the fork should
> do the trick. (Or arrange for subsequent children to
> explicitly close the parent ends of sibling's pipes.)
Yes, one should prevent these fd's from being inherited to
other children forked off later. (I should have thought about
this, since this is an error that caused a colleague of mine
to spend a w behind a debugger).
BUT ... close-on-exec is not the cure ... as the name says it
closes on exec()'s ... but if you only fork(), and then stay
in the forked process (which is not an unlikely scenario)
close-on-exec doesn't do anything!!!
This is obviously a point to keep in mind when trying to handle
multiple concurrent children ... because it cannot be cured right
here, right now.
[color=darkred]
> - close pipe1Rd
> - close pipe2Wr
> If you don't close these extra handles, the parent
> will not see child's stdin/out closes (unless the
> child completely exits and everything closes.)
Yes, true again ... so far (whenever I used this construction) I kept
fd's open util exit()/_exit() was called ... because usually
I want to transfer some kind of final information about the execution
done ...
but generally you're right ... when you don't need the communication
any more (and you want to let the parent know that no more data is to
come) call close() ... even if you have to do some more computaion
in the child ... this way the parent knows, that it doesn't need to
wait for data any more and can proceed. (Don't forget to call wait()
or the family-buddies of wait() later).
Rainer
| |
| Grumble 2005-04-19, 3:59 pm |
| Rainer Temme wrote:
> Grumble wrote:
>
>
> You probably missunderstood what I wrote ...
> You are _not_ using the driver-programs stdin/stdout...
> You are connecting the scripts stdin/stdout to two filedeskriptors
> the driver can write-to and read-from.
> Therefore $ driver 1>somewhere.out 2>somewhere.err 0>somewhere.in
> is still working fine.
I am thoroughly  .
Let us say I run driver 1>stdout.txt 2>stderr.txt
then my program will start with these streams:
keyboard -> driver 1-> stdout.txt
2-> stderr.txt
When I fork/exec a shell script, the script inherits the driver's file
descriptors, i.e.
keyboard -> script 1-> stdout.txt
2-> stderr.txt
In other words, the programs run by the script will write their output
to stdout.txt and stderr.txt
You propose I use the shell script's stdout to communicate back to the
driver. I don't understand how I am supposed to do that, since stdout is
already used to save some information to a file?
Or is there a fundamental misunderstanding on my part?
--
Regards, Grumble
| |
| Alex Fraser 2005-04-19, 8:57 pm |
| "Grumble" <devnull@kma.eu.org> wrote in message
news:d43747$mje$1@news-rocq.inria.fr...
> Rainer Temme wrote:
[snip][color=darkred]
[snip][color=darkred]
> I am thoroughly .
It's Rainer who misunderstood (or missed the point). You seem to understand
perfectly.
Following his advice - using dup2() to connect stdout of the child forked
from the driver to one of the pipes, before exec*() - will make output on
stdout from the commands invoked by the script appear at the driver's end of
the pipe and not (assuming driver invocation as above) stdout.txt.
One way around this would be to use Rainer's suggestion, except with another
descriptor for the script's "return string" instead of stdout. Think of this
as an extension of the 0=stdin/1=stdout/2=stderr convention, followed by
both the driver and the script. For example, in the driver:
pipe(to_script)
pipe(from_script)
fork()
parent:
close(to_script[0])
close(from_script[1])
write(to_script[1], params)
close(to_script[1])
Loop read(from_script[0]) until EOF (implies script finished)
close(from_script[0])
wait()
child:
close(to_script[1])
close(from_script[0])
dup2(to_script[0], 0) /* script still reads params on stdin */
close(to_script[0])
dup2(from_script[1], 999) /* script shall return string on 999 */
close(from_script[1])
exec(script)
(Note that this, like Rainer's original suggestion, assumes the script reads
all parameters before it creates any output. If that isn't the case, you'll
need to use select()/poll() to write the parameters and read anything the
script returns without potentially causing deadlock.)
And, assuming the script is interpreted by bash, in the script:
echo "this goes to driver's stdout, eg stdout.txt"
echo "this string is returned" >&999 # shell does dup2(999, 1)
The only snag is that a command executed by the shell may decide to write to
the "return string" descriptor for no good reason, causing confusion for the
driver. It's highly unlikely to be an issue, but can easily be fixed:
naughty_command params... 999>/dev/null
HTH,
Alex
| |
| Rainer Temme 2005-04-20, 8:58 am |
| Alex Fraser wrote:
> It's Rainer who misunderstood (or missed the point). You seem to understand
> perfectly.
> One way around this would be to use Rainer's suggestion, except with another
> descriptor for the script's "return string" instead of stdout. Think of this
> as an extension of the 0=stdin/1=stdout/2=stderr convention, followed by
> both the driver and the script. For example, in the driver:
Hi Alex,
"aaaahhh, here's the magic cookie" ...
yes, correct, I didn't see, that he wants _different_ streams
for stdout of the script and for a completion message ...
Probably, because I personally wouldn't use this construction...
But for the given case your modifications seem reasonable.
Rainer
|
|
|
|
|