Unix Programming - FILE* behaviour over BSD sockets

Author

FILE* behaviour over BSD sockets - how?

Aurelian Melinte

2004-05-12, 9:08 pm

Hello,

I am looking to make a socket behave like a FILE*.

The problem: I have to use a library (which I cannot modify) with an API reading/writing records to FILE*. The data I have to read comes through a socket. The socket has been set to blocking and I successfully fdopen(socket). However, the API reads once i
n a while incomplete records because the underlying socket calls seem to return as soon as there is a packet to read but not when the whole requested number of bytes has been read.

Do not see any solution...

Regards,
a.

Rich Teer

2004-05-12, 9:08 pm

On Wed, 12 May 2004, Aurelian Melinte wrote:

> The problem: I have to use a library (which I cannot modify) with an API reading/writing records to FILE*. The data I have to read comes through a socket. The socket has been set to blocking and I successfully fdopen(socket). However, the API reads once
in a while incomplete records because the underlying socket calls seem to return as soon as there is a packet to read but not when the whole requested number of bytes has been read.
>
> Do not see any solution...

Use readn(), which specifically doesn't return until the requested
number of bytes has been read (unless there's an error).

Here's one version (I use tab stops of 4), from my forthcoming
book:

------------------------------------------8<---------------------------
#include <unistd.h>
#include <errno.h>

ssize_t readn (int fd, void *buf, size_t num)
{
ssize_t res;
size_t n;
char *ptr;

n = num;
ptr = buf;
while (n > 0) {
if ((res = read (fd, ptr, n)) == -1) {
if (errno == EINTR)
res = 0;
else
return (-1);
}
else if (res == 0)
break;

ptr += res;
n -= res;
}

return (num - n);
}
------------------------------------------8<---------------------------

Enjoy,

--
Rich Teer, SCNA, SCSA

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-online.net

Måns Rullgård

2004-05-12, 9:08 pm

Rich Teer <rich.teer@rite-group.com> writes:

> On Wed, 12 May 2004, Aurelian Melinte wrote:
>
^^^^^^^^^^^^^^^^^^^^^[color=darkred]
>
> Use readn(), which specifically doesn't return until the requested
> number of bytes has been read (unless there's an error).

The OP can't modify the reading function.

--
Måns Rullgård
mru@kth.se

David Schwartz

2004-05-12, 9:08 pm

"Aurelian Melinte" <amelinte@b504-12.info.polymtl.ca> wrote in message
news:zWtoc.16153$Ee6.15427@charlie.risq.qc.ca...

> I am looking to make a socket behave like a FILE*.

> The problem: I have to use a library (which I cannot modify) with an API
> reading/writing records to FILE*. The data I have to read comes through a
> socket. The socket has been set to blocking and I successfully
> fdopen(socket). However, the API reads once in a while incomplete records
> because the underlying socket calls seem to return as soon as there is a
> packet to read but not when the whole requested number of bytes has been
> read.

> Do not see any solution...

The only way I can think of to ensure that a 'read' returns the whole
number of bytes requested is for you to write the socket data to a file
yourself and then tell this API to read from that FILE.

Basically, let the existing code talk to a file, and you use your socket
code to synchronize the data in that file with whatever's on the other end.

DS

Barry Margolin

2004-05-12, 9:30 pm

In article <zWtoc.16153$Ee6.15427@charlie.risq.qc.ca>,
Aurelian Melinte <amelinte@b504-12.info.polymtl.ca> wrote:

> Hello,
>
> I am looking to make a socket behave like a FILE*.
>
> The problem: I have to use a library (which I cannot modify) with an API
> reading/writing records to FILE*. The data I have to read comes through a
> socket. The socket has been set to blocking and I successfully
> fdopen(socket). However, the API reads once in a while incomplete records
> because the underlying socket calls seem to return as soon as there is a
> packet to read but not when the whole requested number of bytes has been
> read.

That's true for just about any type of input stream on Unix. For
instance, if you're reading from a terminal in cooked mode, the
underlying read() call will return as soon as a whole line is entered,
not when the requested number of bytes are available. And a read() from
a pipe will return whatever happens to be available in the pipe buffer.

So I'm not sure why this library has any problems with sockets that it
wouldn't have with most other devices. Only ordinary files will
reliably read as many bytes as you ask for (unless it reaches EOF first).

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

David Schwartz

2004-05-13, 5:32 am

"Barry Margolin" <barmar@alum.mit.edu> wrote in message
news:barmar-FAC6D0.20565212052004@comcast.ash.giganews.com...

> So I'm not sure why this library has any problems with sockets that it
> wouldn't have with most other devices. Only ordinary files will
> reliably read as many bytes as you ask for (unless it reaches EOF first).

It will have problems with most other devices. It was designed to only
handle ordinary files.

The solution is to make sure it only handles ordinary files. The OP
will have to manipulate the network and ordinary files to allow the existing
code to deal with only those ordinary files.

DS

Alexander Krisak

2004-05-13, 5:32 am

294474405
Hello, "Aurelian Melinte"
> Hello,
>
> I am looking to make a socket behave like a FILE*.
>
> The problem: I have to use a library (which I cannot modify) with an API
reading/writing
> records to FILE*. The data I have to read comes through a socket. The
socket has been set to
> blocking and I successfully fdopen(socket). However, the API reads once in
a while incomplete
> records because the underlying socket calls seem to return as soon as
there is a packet to read
> but not when the whole requested number of bytes has been read.
>
> Do not see any solution...
as partial solution you can call ioctl(fileno(f), FIONREAD, &l) to make sure
that at least one record is complete (received), then you can fread() one or
more ready records.

--
chris

Søren Hansen

2004-05-13, 6:31 am

On Wed, 12 May 2004 18:21:19 +0000, Aurelian Melinte wrote:

> The problem: I have to use a library (which I cannot modify) with an
> API reading/writing records to FILE*. The data I have to read comes
> through a socket. The socket has been set to blocking and I successfully
> fdopen(socket). However, the API reads once in a while incomplete records
> because the underlying socket calls seem to return as soon as there is a
> packet to read but not when the whole requested number of bytes has been
> read.

I've never tried using it on sockets, but perhaps fdopen() will do the
trick?

--
Salu2, Søren.

Aurelian Melinte

2004-05-13, 11:40 am

Barry Margolin <barmar@alum.mit.edu> wrote:
> In article <zWtoc.16153$Ee6.15427@charlie.risq.qc.ca>,
> Aurelian Melinte <amelinte@b504-12.info.polymtl.ca> wrote:
>
[color=darkred]
>
> That's true for just about any type of input stream on Unix. For

> not when the requested number of bytes are available. And a read() from
> a pipe will return whatever happens to be available in the pipe buffer.
>
> So I'm not sure why this library has any problems with sockets that it
> wouldn't have with most other devices. Only ordinary files will
> reliably read as many bytes as you ask for (unless it reaches EOF first).
>

The library deals with files - unfortunately. You and David are right: I cannot use a pipe but a a true file only.

Regards

Aurelian Melinte

2004-05-13, 11:40 am

S?ren Hansen <sh@warma.dk> wrote:
> On Wed, 12 May 2004 18:21:19 +0000, Aurelian Melinte wrote:
>
>
> I've never tried using it on sockets, but perhaps fdopen() will do the
> trick?
>

Yes, fdopen succeeds on sockets

Darko M.

2004-05-13, 11:40 am

> > Hello,[color=darkred]

I have verified the behaviour of the fscanf function when it is
waiting for a string from the file pointer.

I made a server side program that executes fscanf( f, "%s", array ),
where the "f" argument is the one got from the fdopen(
socket_descriptor, "r" ).

The client side prints an array from the other side of the connection,
but does not print a separator, then sleeps for 10 seconds, then
prints another array and a separator.

After the first array from the client, the server normally sleeps and
waits for more. After the clients sends the other array AND the
separator, the server wakes up and prints the whole string. That means
that the stdlib works correct.

I thought that perhaps it's something about signals. So I put the
server to wait, the client to send the first string, and then throw a
signal to the server side ( I put the signal handler to execute an
empty block, so the read() would have been interrupted ). Nothing. The
server side sleeps as usual and waits for the rest. It waits until the
separator is found, and the separator only. Not event interrupt from a
signal can stop it ( unless the signal hasn't been handled by a
function that does something that would explicitely interrupt it in a
way ).

This same behaviour is got when it is waiting for a number (I checked
that too).

That gives me the conclusion that your problem is probably about bad
synchronization with the client (with the other side). You probably
send a separator from the other side, and you don't know you did. The
standard library hasn't got (as far as I know) something like
fnscanf(), to read an exact number of characters, so if I'm missing
something in the upper text, than I would suggest you rather use a
file descriptor calls set, than the standard library.

If you're actually not using the standard library functions set,
fscanf() etc. than suggest your programmer to repair the ones you're
using, so they handle interrupts properly etc.

I use Linux 2.4.22. Perhaps it matters, however, if you're using
another version. Although I don't think so, since it's the standard
library we're talking about, and it is STANDARD - as it's name
suggests :)

Jem Berkes

2004-05-13, 12:40 pm

>> The problem: I have to use a library (which I cannot modify) with an
>
> I've never tried using it on sockets, but perhaps fdopen() will do the
> trick?

I had meant to try this myself on an earlier project. I just tried it now
under FreeBSD and it worked... fdopen on the socket with "r+" followed by
fgets and fprintf all worked as intended.

--
Jem Berkes
http://www.sysdesign.ca/

joe@invalid.address

2004-05-13, 12:40 pm

"Alexander Krisak" <chris@imp.lg.ua> writes:

> 294474405
> Hello, "Aurelian Melinte"
> as partial solution you can call ioctl(fileno(f), FIONREAD, &l) to
> make sure that at least one record is complete (received), then you
> can fread() one or more ready records.

The problem with that is that fread() will buffer data, so it could
read one full record and part of another one. So the code consumes the
first full record and leaves the rest in the stdio buffer. Then if
enough data comes in to complete the next record, ioctl() won't tell
you a full record is ready to read. The OP would have to keep track of
how many bytes are available on the socket and combine that with how
many are available in the stdio buffer.

Joe
--
"Surprise me"
- Yogi Berra when asked where he wanted to be buried.

Alexander Krisak

2004-05-13, 1:32 pm

Hello, <joe@invalid.address>
>
> The problem with that is that fread() will buffer data, so it could
> read one full record and part of another one. So the code consumes the
> first full record and leaves the rest in the stdio buffer. Then if
> enough data comes in to complete the next record, ioctl() won't tell
> you a full record is ready to read. The OP would have to keep track of
> how many bytes are available on the socket and combine that with how
> many are available in the stdio buffer.
so, we can disable buffering by setbuf()/setvbuf() function.
--
chris

Barry Margolin

2004-05-13, 1:32 pm

In article <pan.2004.05.13.09.29.54.973678@warma.dk>,
Søren Hansen <sh@warma.dk> wrote:

> On Wed, 12 May 2004 18:21:19 +0000, Aurelian Melinte wrote:
>
>
> I've never tried using it on sockets, but perhaps fdopen() will do the
> trick?

That's precisely what he said he's doing -- see the 4th line of the text
you quoted.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Barry Margolin

2004-05-13, 1:32 pm

In article <%ILoc.16168$Ee6.4948@charlie.risq.qc.ca>,
Aurelian Melinte <amelinte@b504-12.info.polymtl.ca> wrote:

> Barry Margolin <barmar@alum.mit.edu> wrote:
>
>
>
> The library deals with files - unfortunately. You and David are right: I
> cannot use a pipe but a a true file only.

The library has a serious bug, and you should report it to the
developer. Although full-length reads are normal for ordinary files,
I'm not sure that stdio actually guarantees this. There's no excuse for
a general purpose I/O library to expect this from the underlying calls.

The only types of applications that can get away with dealing only with
files are those that perform random access.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

David Schwartz

2004-05-13, 1:32 pm

"Alexander Krisak" <chris@imp.lg.ua> wrote in message
news:2ghlmrF31fenU1@uni-berlin.de...

[color=darkred]
[color=darkred]
> so, we can disable buffering by setbuf()/setvbuf() function.

Sadly, you can still deadlock the network stack. You can't use FIONREAD
to wait until a certain number of byte have been received because the stack
may be waiting for you to read the bytes it has already received before it
will receive any more.

DS

Søren Hansen

2004-05-13, 2:33 pm

Den Thu, 13 May 2004 12:30:30 -0400. skrev Barry Margolin:
> That's precisely what he said he's doing -- see the 4th line of the text
> you quoted.

Hmmm.. So he does. I only read the first 72 characters of his original
message, since he didn't break the lines as he should have. :-) Oh, well,
I'll butt out again.

--
Salu2, Søren.

Alexander Krisak

2004-05-14, 2:31 am

Hello, "David Schwartz"
>
>
>
> Sadly, you can still deadlock the network stack. You can't use
FIONREAD
> to wait until a certain number of byte have been received because the
stack
> may be waiting for you to read the bytes it has already received before it
> will receive any more.

you right, but this method still applicable for records that smaller then
SO_RCVBUF, another problem, that we can't detect, if other side down on
middle
of record (connection broken or peer down), it can be partially solved by
setup
timeout for read operation. Following code tested and worked fine on linux.

;-----------------X8
#include <stdio.h>
#include <string.h>
#include <sys/ioctl.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <arpa/inet.h>

#ifndef min
#define min(a, b) ((a) > (b) ? (b) : (a))
#endif

FILE* sock_fopen(char* host)
{ int fd = -1;
struct sockaddr_in sin;
FILE* f = NULL;

if (host == NULL)
return NULL;

if ((fd = socket(PF_INET, SOCK_STREAM, 0)) == -1)
return NULL;

memset(&sin, 0, sizeof(sin));
sin.sin_family = AF_INET;
sin.sin_port = htons(9876);
sin.sin_addr.s_addr = inet_addr(host);
do
{ if (connect(fd, (struct sockaddr*)&sin, sizeof(sin)) == -1)
break;
f = fdopen(fd, "rb");
} while (0);
if ((f == NULL) && (fd != -1))
close(fd);
else
setvbuf(f, NULL, _IONBF, 0);
return f;
}

int sock_fread(void* buff, size_t size, size_t nmemb, FILE* f, unsigned int
timeout)
{ int l = 0;
unsigned int to = timeout;

while (l < size)
{ if (ioctl(fileno(f), FIONREAD, &l) == -1)
return 0;
if (l < size)
{ sleep(1);
if ((timeout != 0) && (--to == 0))
return 0;
}
}
return fread(buff, size, min(nmemb, l / size), f);
}

int main(int argc, char* argv[])
{ FILE* f;
char buff[20];
int i;

if ((f = sock_fopen("127.0.0.1")) == 0)
{ fprintf(stderr, "sock_fopen()");
return 1;
}

for (i = 0; i < 10; i++)
{ if (sock_fread(buff, sizeof(buff), 1, f, 10) == 0)
break;
else
printf("Have some data for you\n");
}

fclose(f);
return 0;
}
;-----------------X8
--
chris

Sponsored Links