Unix Programming - Fast communication

Author

Fast communication

Tim Frink

2007-11-16, 7:10 pm

Hi,

I need to implement a fast communication between 2
C/C++ programs. A direct communication over memory
(integrating one program into the other) is not possible
to some constraints.

Currently, both applications exchange some decimal numbers
(results for solving some equations) via a file. The first
program creates a file, writes the equations, then invokes
the other program by "system" passing the file as command line
parameter and after the second file has written its results
into a file, this file is finally read by the caller.
For a large number of calls in a loop, the performance suffers
since most time is spent with I/0 handling. Are there
any more efficient ways for communication?

Regards,
Tim

Benoit Lefebvre

2007-11-16, 7:10 pm

Can you call this program, then output the numbers to the STDIN of the
program ?

Something like that: (This is an edited copy/paste from an old thing
that I have done.. it may be missing some stuff)
------------ 8< -----------------------------------------------------
FILE *fp = popen("/path/to/other/program", "w");
fprintf(fp, "my data");
fclose(fp);
------------ 8< -----------------------------------------------------

--Ben

On Nov 16, 12:13 pm, Tim Frink <plfr...@yahoo.de> wrote:
> Hi,
>
> I need to implement a fast communication between 2
> C/C++ programs. A direct communication over memory
> (integrating one program into the other) is not possible
> to some constraints.
>
> Currently, both applications exchange some decimal numbers
> (results for solving some equations) via a file. The first
> program creates a file, writes the equations, then invokes
> the other program by "system" passing the file as command line
> parameter and after the second file has written its results
> into a file, this file is finally read by the caller.
> For a large number of calls in a loop, the performance suffers
> since most time is spent with I/0 handling. Are there
> any more efficient ways for communication?
>
> Regards,
> Tim

William Ahern

2007-11-16, 7:10 pm

Tim Frink <plfriko@yahoo.de> wrote:
> Hi,

> I need to implement a fast communication between 2
> C/C++ programs. A direct communication over memory
> (integrating one program into the other) is not possible
> to some constraints.

What constraints? Are you restricted to only using a file? It would be a
waste of our time to make extensive but useless suggestions....

> Currently, both applications exchange some decimal numbers
> (results for solving some equations) via a file. The first
> program creates a file, writes the equations, then invokes
> the other program by "system" passing the file as command line
> parameter and after the second file has written its results
> into a file, this file is finally read by the caller.
> For a large number of calls in a loop, the performance suffers
> since most time is spent with I/0 handling. Are there
> any more efficient ways for communication?

I'll assume you need to execute the data processor everytime.

How big are the files? You can setup a pipe to stdin of the executed
program, and pass the equations over the pipe. To preserve message
boundaries you can use anonymous unix domain sockets--socketpair(AF_UNIX)--
instead of pipe(). Using twp pipes, or a single socket, you can return data
in the other direction as well.

Stephan Ceram

2007-11-17, 8:08 am

On Fri, 16 Nov 2007 12:48:26 -0800, Benoit Lefebvre wrote:

> Can you call this program, then output the numbers to the STDIN of the
> program ?

Yes, it can.

>
> Something like that: (This is an edited copy/paste from an old thing
> that I have done.. it may be missing some stuff)
> ------------ 8< -----------------------------------------------------
> FILE *fp = popen("/path/to/other/program", "w");
> fprintf(fp, "my data");
> fclose(fp);
> ------------ 8< -----------------------------------------------------

This seems to be the passing data into one direction. How to I
get the information that the called program generates to STDIN?

Tim

Stephan Ceram

2007-11-17, 8:08 am

> What constraints? Are you restricted to only using a file? It would be a
> waste of our time to make extensive but useless suggestions....

The called software is under GPL, the caller not. Thus, mixing
of both source codes must be avoided. That's the only restriction.

>
>
> I'll assume you need to execute the data processor everytime.

Yes, exactly.

>
> How big are the files? You can setup a pipe to stdin of the executed
> program, and pass the equations over the pipe. To preserve message
> boundaries you can use anonymous unix domain sockets--socketpair(AF_UNIX)--
> instead of pipe(). Using twp pipes, or a single socket, you can return data
> in the other direction as well.

The files are rather small. Approx. 10-20 decimals. What would you
prefer for this problem, a single socket or two pipes? And what is
easier to set up?

Thank you.
Tim

Stephan Ceram

2007-11-17, 8:08 am

On Fri, 16 Nov 2007 12:54:15 -0800, William Ahern wrote:

> I'll assume you need to execute the data processor everytime.
>
> How big are the files? You can setup a pipe to stdin of the executed
> program, and pass the equations over the pipe. To preserve message
> boundaries you can use anonymous unix domain sockets--socketpair(AF_UNIX)--
> instead of pipe(). Using twp pipes, or a single socket, you can return data
> in the other direction as well.

What I forgot to ask in my last posting: can you approximate how much
faster communication of small data via sockets/pipes is compared to
a file usage? I assume that the former is performed directly in memory,
while the file requires to access the slow hard disk.

(Please don't be surprised, I'm using the account of a friend for posting
today)

William Ahern

2007-11-17, 7:09 pm

Stephan Ceram <linuxkaffee_@_gmx.net> wrote:
> On Fri, 16 Nov 2007 12:54:15 -0800, William Ahern wrote:

[color=darkred]
> What I forgot to ask in my last posting: can you approximate how much
> faster communication of small data via sockets/pipes is compared to
> a file usage? I assume that the former is performed directly in memory,
> while the file requires to access the slow hard disk.

Well, passing data through a file wouldn't need to touch the disk. It would
all stay in memory. The problem with the file method is that opening a file
is costly, whereas calling socketpair()+dup2() is much cheaper. Passing over
a socket (or pipe, though for small messages I'd use AF_UNIX/SOCK_DGRAM to
keep boundaries) would also likely be slightly faster, but really having to
fork and exec (especially the latter) a process like this would subsume the
difference.

There's a lot of bookkeeping involved with opening and closing files, not so
much reading and writing to them.

Peter J. Holzer

2007-11-17, 7:09 pm

On 2007-11-17 13:03, Stephan Ceram <linuxkaffee_@_gmx.net> wrote:
>
> The called software is under GPL, the caller not. Thus, mixing
> of both source codes must be avoided. That's the only restriction.
>

"system" invokes /bin/sh which in turn invokes the program. Using
fork/exec removes that overhead.
[color=darkred]
>
> Yes, exactly.
>
>
> The files are rather small. Approx. 10-20 decimals.

Then the time to open and read this file is almost certainly dwarfed by
the overhead of starting the program. Try to avoid starting the program
for every set of data. Since you have the source code for the program
you could change it to run in a loop, processing data as it comes in.

hp

--
_ | Peter J. Holzer | It took a genius to create [TeX],
|_|_) | Sy

Alex Fraser

2007-11-17, 7:09 pm

"Tim Frink" <plfriko@yahoo.de> wrote in message
news:pan.2007.11.16.17.13.29.548492@yahoo.de...
> I need to implement a fast communication between 2
> C/C++ programs. A direct communication over memory
> (integrating one program into the other) is not possible
> to some constraints.
>
> Currently, both applications exchange some decimal numbers
> (results for solving some equations) via a file. The first
> program creates a file, writes the equations, then invokes
> the other program by "system" passing the file as command line
> parameter and after the second file has written its results
> into a file, this file is finally read by the caller.
> For a large number of calls in a loop, the performance suffers
> since most time is spent with I/0 handling. Are there
> any more efficient ways for communication?

Are you certain the "I/O handling" is the dominant factor in performance?
Calling system() generally starts a shell to interpret the command, and I
think starting a shell is likely to take significantly longer than the "I/O
handling", given (as you said elsewhere in the thread) that the files are
small.

It depends on the system() call, but using fork()/exec() to start the other
application and waiting for it to finish is likely to be quite
straightforward. You'll have to do this anyway if you want to follow William
Ahern's suggestion of avoiding the files by using a socket or pair of pipes,
so I would suggest trying it first in case I'm right above :).

Avoiding the files is also fairly straightforward /provided/ the other
application reads all of its input before generating any output. If not, you
need significant extra code (eg non-blocking IO and a select() or poll()
loop) to avoid the possible deadlock where the other application is blocked
writing output to you while you are blocked writing input to it.

Alex

Sponsored Links