Unix Programming - advices on sockets

Author

advices on sockets

mast4as

2008-03-13, 7:52 pm

Hi everyone,

Yes I spend quite some time (2 days) reading tutorials, man pages on
sockets and related topics but still can't seem to find the proper way
of doing what I need. It's not so much that it's impossible, it's just
that I don't have the knowledge and s

over people's experiences on
this topic ;-)

The little app I need to write is a x-window to which a graphic
program sends an image. The graphic program renders a frame and send
it to the x-window in tiles (blocks of pixels). I ended up deciding to
implement that with a server (x-window) -client (graphic app) type of
model. The reason is that eventually several "instances" of the
graphic program can run on the computer at the same time and the
images that each "instance" works on, needs to be send to the same
window.

The other condition is that the x-window stays active even though the
graphic app has finished processing the frame. The user can therefore
continue manipulating the frame in the x-window (zooming, panning.
etc...)

Now I must say I am trying to prototype things here, some I just want
to find a simple, elegant and robust way to do that. I am not a socket
expert (obviously), have very modest programming skills in comparison
to you all... So I am just really asking for a bit of supervision
here, with something I can't seem to find any answers for in posts
tutorials on the web. So please be kind ;-)

I have many question regarding this problem...

1/ on the client side (the graphic app), I'd like to start to the
server (the x-window) if it's not running yet. This yields 2 sub-
questions:
1.1 what's the best way to find out that the server is not running ?
Is that fact
that a call to connect() fails is enough ?
1.2 is there a way I can start my server/x-window code by another
mean that
calling the function exec() ? I am asking that question because it
means the
code for the server has to be compiled as a separate app. Can I
avoid that?
Can I write the code for the x-window/server with the code of the
client and
start the x-window from within the graphics app as a *separate
process* that
won't die when the graphics app has finished processing the image ?

2/ my other problem (thing I am not clear about) is the way I am
passing the data from the client (graphics app) and the server (x-
window). I have also 2 questions for this.
2.1 ok i read in the docs that by default, accept() is blocking.
Remember that I
wrote at the beginning of the post that I want the x-window app to
run as an
independent process from the graphics app and that the user can
manipulate
the frame in the x-window using shortkeys for example (z = zoom in,
Z = zoom
out, or 'q' to quit the x-window program). So the x-display/server
program needs
to check for both new data coming through the socket but also X
events. If I use
a loop to check for new connections from clients, because accept()
is blocking,
it stops me from checking X-events.

// this for example doen't work because accept is blocking
while(1) {
checkXEvent(&c);
if (c=='q')
quit = 1;
if (accept(sockfd, ...)==-1)
continue;
// we have an incoming connection with a client, proceed..., fork
to create a
// a process child and treat packed of incoming data
...
}

so my question here is the following. Is the only way to do this is
by making the
socket non-blocking ? But in that case, isn't the fact that having a
non-blocking
socket uses a lot of unnecessary CPU resources (because it never
stops
checking if there's an incoming connection).

I played with select() (commented part of my code) but it didn't
work. In particular
my understanding is that if I use select() is can't use the fork
technique
anymore. In other words if 2 instances of the graphics app processing
2
images are running at the same time, the file descriptor set will
contain
2 file descriptors (one for each connection it has had from the 2
running clients).
Meaning if the 2 clients send data to the sever at the same time, I
will have
to read the data from each client and do i need to do with that data
sent.
Is that a better way than fork ?

2.2 to pass the data i didn't find anything better that coming up
with a stupid
type of protocol. I send a request to the server first that tells it
what it should
expect to read next (some info about the size of the incoming tile,
or the RGB
data for the tile). In the little prototype I worked on it seems to
work but I am not
sure it's reliable ? Is it how you would do it ? Is there a better
way ?

I put the code online with you have the patience and the kindness to
have a look

http://www.scratchapixel.com/docs/proto.cpp // sim graphics app
http://www.scratchapixel.com/docs/server.cpp // sim server/x-window

I really apologize for that long post, but hopefully it will have some
interesting answers and will help me and other people in the future,
as a part from basic client-server code examples that deals with
buff[1024], I didn't find anything else out there that answered
clearly those questions.

Thank you so much -coralie

Barry Margolin

2008-03-13, 7:52 pm

In article
<c59bd765-7adc-4e11-a022-189e4433dfd0@s37g2000prg.googlegroups.com>,
mast4as <mast4as@yahoo.com> wrote:

> Hi everyone,
>
> Yes I spend quite some time (2 days) reading tutorials, man pages on
> sockets and related topics but still can't seem to find the proper way
> of doing what I need. It's not so much that it's impossible, it's just
> that I don't have the knowledge and s

over people's experiences on
> this topic ;-)
>
> The little app I need to write is a x-window to which a graphic
> program sends an image. The graphic program renders a frame and send
> it to the x-window in tiles (blocks of pixels). I ended up deciding to
> implement that with a server (x-window) -client (graphic app) type of
> model. The reason is that eventually several "instances" of the
> graphic program can run on the computer at the same time and the
> images that each "instance" works on, needs to be send to the same
> window.
>
> The other condition is that the x-window stays active even though the
> graphic app has finished processing the frame. The user can therefore
> continue manipulating the frame in the x-window (zooming, panning.
> etc...)
>
> Now I must say I am trying to prototype things here, some I just want
> to find a simple, elegant and robust way to do that. I am not a socket
> expert (obviously), have very modest programming skills in comparison
> to you all... So I am just really asking for a bit of supervision
> here, with something I can't seem to find any answers for in posts
> tutorials on the web. So please be kind ;-)
>
> I have many question regarding this problem...
>
> 1/ on the client side (the graphic app), I'd like to start to the
> server (the x-window) if it's not running yet. This yields 2 sub-
> questions:
> 1.1 what's the best way to find out that the server is not running ?
> Is that fact
> that a call to connect() fails is enough ?

Yes.

> 1.2 is there a way I can start my server/x-window code by another
> mean that
> calling the function exec() ? I am asking that question because it
> means the
> code for the server has to be compiled as a separate app. Can I
> avoid that?
> Can I write the code for the x-window/server with the code of the
> client and
> start the x-window from within the graphics app as a *separate
> process* that
> won't die when the graphics app has finished processing the image ?

Yes. You could write it as a subroutine in your program, and use fork()
to start a new process. In the new process you would call this
subroutine, while the original process would go back to trying to
connect.

You might also want to arrange some additional communication between the
parent and child process, so that the child can tell the parent when it
has started the server. Otherwise, if the parent tries to connect too
soon, it might think it needs to start the server again. You could use
a semaphore for this.

>
> 2/ my other problem (thing I am not clear about) is the way I am
> passing the data from the client (graphics app) and the server (x-
> window). I have also 2 questions for this.
> 2.1 ok i read in the docs that by default, accept() is blocking.
> Remember that I
> wrote at the beginning of the post that I want the x-window app to
> run as an
> independent process from the graphics app and that the user can
> manipulate
> the frame in the x-window using shortkeys for example (z = zoom in,
> Z = zoom
> out, or 'q' to quit the x-window program). So the x-display/server
> program needs
> to check for both new data coming through the socket but also X
> events. If I use
> a loop to check for new connections from clients, because accept()
> is blocking,
> it stops me from checking X-events.
>
> // this for example doen't work because accept is blocking
> while(1) {
> checkXEvent(&c);
> if (c=='q')
> quit = 1;
> if (accept(sockfd, ...)==-1)
> continue;
> // we have an incoming connection with a client, proceed..., fork
> to create a
> // a process child and treat packed of incoming data
> ...
> }
>
> so my question here is the following. Is the only way to do this is
> by making the
> socket non-blocking ? But in that case, isn't the fact that having a
> non-blocking
> socket uses a lot of unnecessary CPU resources (because it never
> stops
> checking if there's an incoming connection).

You need to use select() to listen for activity on multiple sockets at
once. Or you can use threads, where one thread waits for new
connections, and other threads process existing connections. If you
only expect a small number of clients this can be an easier way to
program it.

>
> I played with select() (commented part of my code) but it didn't
> work. In particular
> my understanding is that if I use select() is can't use the fork
> technique
> anymore. In other words if 2 instances of the graphics app processing
> 2
> images are running at the same time, the file descriptor set will
> contain
> 2 file descriptors (one for each connection it has had from the 2
> running clients).
> Meaning if the 2 clients send data to the sever at the same time, I
> will have
> to read the data from each client and do i need to do with that data
> sent.
> Is that a better way than fork ?

This is where threads may make things easier. When a connection comes
in, start a new thread that reads from that client.

>
> 2.2 to pass the data i didn't find anything better that coming up
> with a stupid
> type of protocol. I send a request to the server first that tells it
> what it should
> expect to read next (some info about the size of the incoming tile,
> or the RGB
> data for the tile). In the little prototype I worked on it seems to
> work but I am not
> sure it's reliable ? Is it how you would do it ? Is there a better
> way ?

For a simple application like this, your "stupid protocol" may be best.

>
> I put the code online with you have the patience and the kindness to
> have a look
>
> http://www.scratchapixel.com/docs/proto.cpp // sim graphics app
> http://www.scratchapixel.com/docs/server.cpp // sim server/x-window
>
> I really apologize for that long post, but hopefully it will have some
> interesting answers and will help me and other people in the future,
> as a part from basic client-server code examples that deals with
> buff[1024], I didn't find anything else out there that answered
> clearly those questions.
>
> Thank you so much -coralie

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE don't copy me on replies, I'll read them in the group ***

mast4as

2008-03-13, 7:52 pm

Thank you very much for your answer.

> Yes. =A0You could write it as a subroutine in your program, and use fork()=

> to start a new process. =A0In the new process you would call this
> subroutine, while the original process would go back to trying to
> connect.

ok but wouldn't the child process (the x-window/server) die when the
parent (graphics app/client) process dies too ?
That was my understanding of fork() behavior.

> You might also want to arrange some additional communication between the
> parent and child process, so that the child can tell the parent when it
> has started the server. =A0Otherwise, if the parent tries to connect too
> soon, it might think it needs to start the server again. =A0You could use
> a semaphore for this.

hum i never studied semaphore ;-( but will have a look.

> You need to use select() to listen for activity on multiple sockets at
> once. =A0Or you can use threads, where one thread waits for new
> connections, and other threads process existing connections. =A0If you
> only expect a small number of clients this can be an easier way to
> program it.

but isn't select() a blocking call too if the socket is non-blocking ?
I tried that approach as a matter of fact and did a call to select
within the loop that check for x-events as well... something like
that:

while(1) {
checkForXevents(&c);
if (c=3D=3D'q')
break;
// hangs there... wait for something to happen ?
select(fdmax+1...);
}

if I don't make the socket non-blocking the loop hangs on select. If I
make it non-blocking and set a timeout value, you need to wait for a
timeout before the loop continues anyway... which is not so

when
you want no delay in the user interaction with the window (keyboard,
mouse...). I looked at this problem on different angle and really
can't seem to figure it out. Maybe threading as you said would be an
option but that means a thread to catch x-events and treat them, and
other threads to catch new connections and read from those
connections.

When you say using thread are you thinking of pthread or fork ?

I guess what I could do would be the following:

1/ open the display
2/ create a thread to catch x-event and manipulate the framebuffer
accordingly (zoom in...)
3/ create a thread that "accept" new connections
4/ each connection starts a thread that listen to what's coming in on
that connection

It would be ok but I was hoping for something simpler... I don't want
that x-window/server app to use too much memory and CPU cycles to
leave as much processing power for the graphics app...

Thanks for more of your feedbacks/ideas if you have some

-c

Brice Rebsamen

2008-03-13, 10:17 pm

> 1/ open the display
> 2/ create a thread to catch x-event and manipulate the framebuffer
> accordingly (zoom in...)
> 3/ create a thread that "accept" new connections
> 4/ each connection starts a thread that listen to what's coming in on
> that connection
>
> It would be ok but I was hoping for something simpler... I don't want
> that x-window/server app to use too much memory and CPU cycles to
> leave as much processing power for the graphics app...

Why not use a UDP based message passing. In that way you don't have to
handle several threads for each connection. UDP is connectionless.
- the x app opens a socket and listen on it for messages
- the graphic programs send messages to that socket
- the x app processes incoming messages one by one
That MAY do the job in a simple way.

If all your progs run on the same machine you can also consider using
a message queue:
http://www.cs.cf.ac.uk/Dave/C/node25.html

mast4as

2008-03-14, 4:44 am

> Why not use a UDP based message passing. In that way you don't have to
> handle several threads for each connection. UDP is connectionless.
> - the x app opens a socket and listen on it for messages
> - the graphic programs send messages to that socket
> - the x app processes incoming messages one by one
> That MAY do the job in a simple way.

yes possibly but can you manage multiple connections from clients with
UDP ? I guess in that case you need to encode in the data passed from
which 'client' the packet of data is coming from. If for example,
there's 2 graphic apps running, one is sending data for frame #1 and
the other from frame #2, each packet send should start with a header
that needs to be extracted from the data, that would indicate which
frame the data belong to.

Is that what you had in mind ?

> If all your progs run on the same machine you can also consider using
> a message queue:http://www.cs.cf.ac.uk/Dave/C/node25.html

It could and that's what I am doing at the moment, but ideally in the
long term, you could imaging the graphic app running on a remote
machine and sending the result to your screen...

Thanks -c

Barry Margolin

2008-03-15, 4:59 am

In article
<8448b519-b3b6-4505-9348-f42bc62cbed8@a1g2000hsb.googlegroups.com>,
mast4as <mast4as@yahoo.com> wrote:

> Thank you very much for your answer.
>
>
>
> ok but wouldn't the child process (the x-window/server) die when the
> parent (graphics app/client) process dies too ?
> That was my understanding of fork() behavior.

No, child processes are not dependent on the parent.

>
>
> hum i never studied semaphore ;-( but will have a look.
>
>
> but isn't select() a blocking call too if the socket is non-blocking ?
> I tried that approach as a matter of fact and did a call to select
> within the loop that check for x-events as well... something like
> that:
>
> while(1) {
> checkForXevents(&c);
> if (c=='q')
> break;
> // hangs there... wait for something to happen ?
> select(fdmax+1...);
> }
>
> if I don't make the socket non-blocking the loop hangs on select. If I
> make it non-blocking and set a timeout value, you need to wait for a
> timeout before the loop continues anyway... which is not so

when
> you want no delay in the user interaction with the window (keyboard,
> mouse...). I looked at this problem on different angle and really
> can't seem to figure it out. Maybe threading as you said would be an
> option but that means a thread to catch x-events and treat them, and
> other threads to catch new connections and read from those
> connections.

When you use select(), you have to put ALL the sockets and other
descriptors you're using into the fdset, so that you can wait for
activity on any of them.

But I forgot that you're waiting on X events as well, which aren't
descriptors. I'm not an X programmer, but I think there's a way to make
the X event loop (your checkForXevents() function) also check for input
on sockets.

>
> When you say using thread are you thinking of pthread or fork ?

pthread. Using fork is tricky if they're all trying to send to the same
X window.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE don't copy me on replies, I'll read them in the group ***

mast4as

2008-03-15, 4:59 am

> When you use select(), you have to put ALL the sockets and other
> descriptors you're using into the fdset, so that you can wait for
> activity on any of them.
>
> But I forgot that you're waiting on X events as well, which aren't
> descriptors. =A0I'm not an X programmer, but I think there's a way to make=

> the X event loop (your checkForXevents() function) also check for input
> on sockets.

That's it. Actually it's weird i didn't make a search on the web
before with the "XNextEvent socket" because I came with lots of good
answer. For example i found this code which is doing what I need:

http://www.cs.washington.edu/resear...e_patterns/sou=
rce_code/kdesu.src/classes/kdesu'.html

You are right, you can use select for both the sockets and the X11
display which is treated like any other standard file descriptor.

Thank you so much for playing the game. I would have never came across
this idea otherwise. I will post the src of my display program for
future reference once I get it working.

many thanks again -c

mast4as

2008-03-17, 4:41 am

So here is as promised the solution i came up with... i removed lines
which are not related to the problem i was trying to solve... although
there's actually a few other similar examples on the net. It suffices
to do a search withe the right keywords (like XNextEvent + socket)

int main(int argc, char **argv)
{
int sockfd, yes = 1, maxfd;
if ((sockfd = socket(PF_INET, SOCK_STREAM, 0)) < 0) {
fprintf(stderr, "can't create socket");
return -1;
}
....

maxfd = sockfd;

...

int xconnectionfd = initXconnection();
if (xconnectionfd > maxfd)
maxfd = xconnectionfd;

...
// main loop
fd_set active_fds, tmp_fds;
FD_ZERO(&active_fds);
FD_SET(sockfd, &active_fds);
if (xconnectionfd != -1)
FD_SET(xconnectionfd, &active_fds);
int YES = 1;
while (YES) {
tmp_fds = active_fds;
if (select(maxfd+1, &tmp_fds, NULL, NULL, NULL) < 0) {
if (errno == EINTR)
continue;
fprintf(stderr, "failed on select");
exit(1);
}
for (int i = 0; i <= maxfd; ++i) {
if (!FD_ISSET(i, &tmp_fds))
continue;
if (i == xconnectionfd) {
//printf("xevent");
XEvent event_return;
while(XEventsQueued(dpy, QueuedAfterFlush)) {
XNextEvent(dpy, &event_return);
switch(event_return.type) {
case Expose:
break;
case KeyPress:
char c;
XLookupString((XKeyEvent *)&event_return, &c, 1, NULL,
NULL);
if (c == 'q')
YES = 0;
break;
default:
break;
}
}
}
else if (i == sockfd) {
int newfd;
struct sockaddr_in client; // server address
socklen_t addrlen = sizeof client;
if ((newfd = accept(sockfd, (struct sockaddr *)&client,
&addrlen)) < 0) {
fprintf(stderr, "failed on accept");
continue;
}
FD_SET(newfd, &active_fds);
if (newfd > maxfd)
maxfd = newfd;
printf("selectserver: new connection from %s on socket %d\n",
inet_ntoa(client.sin_addr), newfd);
}
else {
// handle data from a client ?
...
}
}
}
close(sockfd);
closeXconnection();
return 0;
}

jason.cipriani@gmail.com

2008-03-17, 8:17 am

On Mar 13, 11:32 am, mast4as <mast...@yahoo.com> wrote:
[snip]
> The little app I need to write is a x-window to which a graphic
> program sends an image. The graphic program renders a frame and send
> it to the x-window in tiles (blocks of pixels). I ended up deciding to
> implement that with a server (x-window) -client (graphic app) type of
> model. The reason is that eventually several "instances" of the
> graphic program can run on the computer at the same time and the
> images that each "instance" works on, needs to be send to the same
> window.
[snip]
> 1/ on the client side (the graphic app), I'd like to start to the
> server (the x-window) if it's not running yet. This yields 2 sub-
> questions:
[snip]

It sounds to me like you only intend to run the "server" and "clients"
on the same computer. Is this correct? If this is what you intend, why
not just have your application display an X window, and then start
multiple threads, each that render part of the graphic? Then you don't
have to deal with sockets at all.

Also note that unless you are are spending time waiting on hardware to
do something and you want to do some other processing in the mean
time, you won't gain any performance by starting more "clients" than
the number of cpu cores you have in your machine. One advantage of
doing it with sockets, like you say, is that you do have the
possibility of running the clients on other computers. But, again, if
your intention is only to do this on a single machine, you're better
off just doing it with a single process containing multiple threads.

Jason

jason.cipriani@gmail.com

2008-03-17, 8:17 am

On Mar 17, 6:54 am, "jason.cipri...@gmail.com"
<jason.cipri...@gmail.com> wrote:
> On Mar 13, 11:32 am, mast4as <mast...@yahoo.com> wrote:
> [snip]> The little app I need to write is a x-window to which a graphic
> [snip]
>
> [snip]
>
> It sounds to me like you only intend to run the "server" and "clients"
> on the same computer. Is this correct? If this is what you intend, why
> not just have your application display an X window, and then start
> multiple threads, each that render part of the graphic? Then you don't
> have to deal with sockets at all.
>

P.S. Dropping the idea of doing it with sockets right off the bat and
just sticking to threads has the advantage of still setting you up to
use sockets again *later* if you do decide you want to run the clients
on more than one machine. Your application, if properly designed,
would not care how each thread rendered it's part of the graphic, so
it is still 100% doable to make some threads connect to other
computers and have the other computers render the image parts. Doing
it this way also swaps what you consider to be the "server" and
"client". In this case, the servers would actually be the remote
machines doing the rendering -- the service they provide is that they
render a part of a graphic for a client. Your main display application
would consist of multiple clients, each connecting to a server on a
remote machine, sending it whatever info it needs to render the image,
and then waiting for a response.

Jason

> Also note that unless you are are spending time waiting on hardware to
> do something and you want to do some other processing in the mean
> time, you won't gain any performance by starting more "clients" than
> the number of cpu cores you have in your machine. One advantage of
> doing it with sockets, like you say, is that you do have the
> possibility of running the clients on other computers. But, again, if
> your intention is only to do this on a single machine, you're better
> off just doing it with a single process containing multiple threads.

mast4as

2008-03-18, 7:20 pm

> > It sounds to me like you only intend to run the "server" and "clients"[color=darkred]

Well that's how it worked at the beginning. The main advantage of
having a socket system is that you can render a frame, display the
result in the xwindow, then start another frame later, and get the
second image displayed in the same window. Eventually if you have kept
the result in the memory of the xwindow app, you can swap between the
2. The only way i have found to get that to work, was by making 2
separate applications and have them communicate by sockets. This way
when the graphic app is done processing a frame (and exit), the
xwindow stays on the screen and is ready for the next process which
you might start by running the graphic app again.

For now, I am running it on the same computer you are right, because I
am testing it on my computer, but if that project evolves the way it
should i might consider using it to display the result of a frame
which is net-rendered by several machines connected by a network. The
server becomes your local computer, eventually, and dispatch the work
to the clients... once bits of images are rendered by the clients they
are displayed to the screen.

Again if you think threads are a better way, I am interested in
looking into that. I am particularly fond of sockets. In particular if
it works well like the prototype i wrote seems to show, it doesn't
seem to me the most reliable technique. Even though I have seen it
running in many places without any problem at all.

If you more suggestion or thinl i can implement what i need with
threads, please let me know.

thanks -c

Sponsored Links