Desperately need help in Tcl socket and fileevent puzzle...

Hi all,
I have been pulling my hair and I still couldn't figure out what was
wrong with my TCL program. I really would appreciate any help/comments
you can give:

I have a small program in TCL that:
1) connect to a socket
2) setup a fileevent handler to read binary data coming from this
socket
When binary data received
2a) The handler does:  set binStream [read $channel]
2a) the handler then appends this "binStream" to a file:
puts -nonewline $binFile $binStream
2b) it then calls a proc to process the data using a copy of this
"binStream":
set binStreamCopy $binStream
processData

3) both "binStreamCopy" and "binStream" are global
4) when "processData" was called, it uses "binary scan" to
process/parses data
from "binStreamCopy"

Looking at the content of the binary file logged in (2a), the content
was 100% correct. However, the proc that processes the data (4) got
corrupted data from time to time from the result of "binary scan" when
it parsed "binStreamCopy".

What did I do wrong? I have been searching around on the web and just
couldn't figure it out.
I would appreciate all your help. Thank you very much in advance.

-Mike-

johndoe_rider@yahoo.com (Mike) wrote:
# Hi all,
# I have been pulling my hair and I still couldn't figure out what was
# wrong with my TCL program. I really would appreciate any help/comments
# you can give:
#
# I have a small program in TCL that:
# 1) connect to a socket
# 2) setup a fileevent handler to read binary data coming from this
# socket
#    When binary data received
#     2a) The handler does:  set binStream [read $channel]

You use fconfigure $channel -translation binary ?

# Looking at the content of the binary file logged in (2a), the content
# was 100% correct. However, the proc that processes the data (4) got
# corrupted data from time to time from the result of "binary scan" when
# it parsed "binStreamCopy".

Can you isolate the corruption to specific bytes? Things like \x0D
becomes \x0A or \x00 to \xFF\xFE?

--
SM Ryan http://www.rawbw.com/~wyrmwif/
OOOOOOOOOO! NAVY SEALS!

Mike wrote:
> I have a small program in TCL that:
> 1) connect to a socket
> 2) setup a fileevent handler to read binary data coming from this
> socket

Did you remember to put:
fconfigure $channel -translation binary
in there?  Without that, it won't work.

>    When binary data received
>     2a) The handler does:  set binStream [read $channel]
>     2a) the handler then appends this "binStream" to a file:
>              puts -nonewline $binFile $binStream

You'll want to also make the $binFile channel a binary channel.

> Looking at the content of the binary file logged in (2a), the content
> was 100% correct. However, the proc that processes the data (4) got
> corrupted data from time to time from the result of "binary scan" when
> it parsed "binStreamCopy".

Ah, let me guess.  You're running on Windows?  Or possibly OSX?

The problem is end-of-line translation, and putting in suitable
[fconfigure] invokations (see above for an example) will fix it (and a
few other things that you've luckily not hit so far). If you're
processing the data in a command written in C, there are a few other
issues you might want to bear in mind, but you don't seem to be
indicating that that's the case here.

Donal.

Thank you for your comment and I appreciate it.

"Donal K. Fellows" <donal.k.fellows@man.ac.uk> wrote in message news:<cg1u9q$v0q$1@godfrey.
mcc.ac.uk>...
> Mike wrote: 
>
> Did you remember to put:
>    fconfigure $channel -translation binary
> in there?  Without that, it won't work.

Yes, I did. The channel was configured as:

fconfigure $binChannel -translation binary -encoding binary -blocking
0

then:

I setup a handler with fileevent to read from this channel.
I'm sorry that I forgot to mention the platform:

This TCL program was running on a Sun/Solaris machine. The reading of
this binChannel was fine as I checked the binary file logged and it
was perfect.

My questions are (in addition to my original post):

- I would believe that my handler can call a Tcl proc like what I'm
doing?
- When data is available at the socket, my handler was called and it
then call
the "processData" proc. What happens if "processData" proc is still
processing
while new data is coming to the socket? Will my handler get called
right away
(interrupting the proc "processData") or will my handler get called
only after
"processData" is done and returned?  I have tried both case and it
didn't
seem to matter as I always got corrupted data in "processData" but
not
in the logged file.

TO make things more clear to you and I would really appreciate all
comments as
I'm really running out of idea, this is the mechanism:

------------------------------
1) open socket, fconfigure it to binary as I mentioned above, then use
fileevent to connect the handler "binReader" to this socket. This
socket
is incoming only (there are only data coming in but not going
out)

2) Waiting for data
3) When data come in, "binReader" got called
3a)  set binStream [read $channel]
2a) the handler then logged/appended this "binStream" to a file:
puts -nonewline $binFile $binStream
2b) it then calls a proc to process the data using a copy of this
"binStream":
set binStreamCopy $binStream
processData

"binStream" and "binStream1" are global variables.
I have tried to pass "binStream" as a parameter to "processData"
but it's still the same: got corrupted data from time to time.
I have also tried to setup a flag so that "binReader" won't call
"processData" if it's not done but I still got the same corrupted
data.
Not everything was corrupted but it happens to some bytes inside
"binStreamCopy":

set doneProcess 1

in binReader:

set binStream [read $channel]
puts -nonewline $binFile $binStream
if {$doneProcess == 1} {
set binStreamCopy $binStream
set doneProcess 0
processData
}

in processData:

global doneProcess
process binStream1 <-- parse binStream1 with binary scan and other
stuffs
set doneProcess 1
return }

Thanks for your input and I appreciate it.

SM Ryan <wyrmwif@tango-sierra-oscar-foxtrot-tango.fake.org> wrote in message news:<10i8cbfh
g1sl32d@corp.supernews.com>...
>
> You use fconfigure $channel -translation binary ?

yes, I did. It was fconfigure with:

fconfigure $binChannel -translation binary -encoding binary -blocking 0


> # Looking at the content of the binary file logged in (2a), the content
> # was 100% correct. However, the proc that processes the data (4) got
> # corrupted data from time to time from the result of "binary scan" when
> # it parsed "binStreamCopy".
>
> Can you isolate the corruption to specific bytes? Things like \x0D
> becomes \x0A or \x00 to \xFF\xFE?

Good question: when I got the corrupted data, it was when I expected
an integer with a value between 4 and 2000 converted with "binary scan",
I got a huge number that cause "processData" to crash so it might be.
However, I have absolutely no error on the logged file. What does it
mean then? I would appreciate any inputs from you.
Thanks.

-mike-

Mike wrote:

> Looking at the content of the binary file logged in (2a), the content
> was 100% correct. However, the proc that processes the data (4) got
> corrupted data from time to time from the result of "binary scan" when
> it parsed "binStreamCopy".

Let me hazard a guess.  TCP/IP only guarantees that the bytes will
arrive in order, it does *not* guarantee that they will arrive in chunks
related in any way to the chunks that you sent.  The collected chunks in
the file show that you're getting the right bytes in the right order,
but any given chunk could contain, for instance, the last few bytes of
an integer, a few whole data fields, and the first few bytes of another
data field.  Just *assuming* you have whole data fields and trying to
parse them with [binary scan] won't always work (but often will, giving
a false sense of security).  You need a layer of logic (based on your
knowledge of the structure of data stream) to decide when you have
enough data to process reliably; I generally rely on a "command code" or
"message length" field to tell me when I've got a whole packet.

Here's an example, where each "command" has an 8-byte header giving a
command code and message length.  If you don't have all the pieces, you
can't go on to the next step:

proc readConnection {chan} {
upvar #0 BUFFER($chan) data
append data [read $chan]
if {[eof $chan]} {
close $chan
return
}
# Do we at least have a complete header?
while {[string length $data] > 7} {
# Extract command code and message length.
binary scan $data "II" cmd len
# Do we have a complete message?
if {[string length $data] < [expr $len + 8]} {return}
# Yes, remove it from buffer.
set msg [string range $data 8 [expr $len + 7]]
set data [string range $data [expr $len + 8] end]
# Process message here...
}
}

To reiterate: just because you sent the data in N chunks with certain
boundaries does *not* mean it will be received in N chunks, or with the
same boundaries.  You must be prepared to handle more or fewer chunks,
and to process the data based on boundaries detectable within the data
stream.

In article <ba1d209d.0408190904.56e915b@posting.google.com>,
Mike <johndoe_rider@yahoo.com> wrote:
>
>Good question: when I got the corrupted data, it was when I expected
>an integer with a value between 4 and 2000 converted with "binary scan",
>I got a huge number that cause "processData" to crash so it might be.
>However, I have absolutely no error on the logged file. What does it
>mean then? I would appreciate any inputs from you.
>Thanks.

Your problem is most likely due your "read" returned a partial chunk
of the multi-byte integer you are expecting.  This is the expected
behavior of an async read request.  There is no guaranty that a single
read will return all the bytes you need.

You would not see this error in the log file as the log file is viewed
later when subsequent reads completes the integer chunk.  But, you can
flush your log immediately when the error occurs and exit your program.
Then examine your log file, you will probably see why the error is
happening.

You need to figure out a way to detect the integer is completely read
before processing it.

-- Jiang Wu
jwu@cs.stanford.edu

Thank you all! I truly appreciate all your helps and suggestions
(I'm a C programmer and this is my 1st TCL program to handle a task
using socket).
Special thanks go to Jiang Wu and John Seal as what they said
was exactly what happened. I, stupidly assumed, that if the sender
sends out x chunks of y size then my TCL will get exactly that and
I now know that's not true. The incoming stream does have a total size
in the 1st 4 bytes of the stream (that I didn't use) so I changed my
"binReader" handler to read this 4-byte and then build up all incoming
data
until all expected bytes are in, before passing them to my
"processData" proc
and voila, everything works!

I sincerely thank all of you.See you!

-Mike-

jwu@xenon.Stanford.EDU (Jiang Wu) wrote in message news:<cg2p27$sj6$1@xenon.Stanford.EDU>..
.
> In article <ba1d209d.0408190904.56e915b@posting.google.com>,
> Mike <johndoe_rider@yahoo.com> wrote: 
>
> Your problem is most likely due your "read" returned a partial chunk
> of the multi-byte integer you are expecting.  This is the expected
> behavior of an async read request.  There is no guaranty that a single
> read will return all the bytes you need.
>
> You would not see this error in the log file as the log file is viewed
> later when subsequent reads completes the integer chunk.  But, you can
> flush your log immediately when the error occurs and exit your program.
> Then examine your log file, you will probably see why the error is
> happening.
>
> You need to figure out a way to detect the integer is completely read
> before processing it.
>
> -- Jiang Wu
>    jwu@cs.stanford.edu

Thank you all! I truly appreciate all your helps and suggestions
(I'm a C programmer and this is my 1st TCL program to handle a task
using socket).
Special thanks go to Jiang Wu and John Seal as what they said
was exactly what happened. I, stupidly assumed, that if the sender
sends out x chunks of y size then my TCL will get exactly that and
I now know that's not true. The incoming stream does have a total size
in the 1st 4 bytes of the stream (that I didn't use) so I changed my
"binReader" handler to read this 4-byte and then build up all incoming
data
until all expected bytes are in, before passing them to my
"processData" proc
and voila, everything works!

I sincerely thank all of you.See you!

-Mike-

jwu@xenon.Stanford.EDU (Jiang Wu) wrote in message news:<cg2p27$sj6$1@xenon.Stanford.EDU>..
.
> In article <ba1d209d.0408190904.56e915b@posting.google.com>,
> Mike <johndoe_rider@yahoo.com> wrote: 
>
> Your problem is most likely due your "read" returned a partial chunk
> of the multi-byte integer you are expecting.  This is the expected
> behavior of an async read request.  There is no guaranty that a single
> read will return all the bytes you need.
>
> You would not see this error in the log file as the log file is viewed
> later when subsequent reads completes the integer chunk.  But, you can
> flush your log immediately when the error occurs and exit your program.
> Then examine your log file, you will probably see why the error is
> happening.
>
> You need to figure out a way to detect the integer is completely read
> before processing it.
>
> -- Jiang Wu
>    jwu@cs.stanford.edu

John Seal wrote:
> To reiterate: just because you sent the data in N chunks with certain
> boundaries does *not* mean it will be received in N chunks, or with the
> same boundaries.  You must be prepared to handle more or fewer chunks,
> and to process the data based on boundaries detectable within the data
> stream.

If you're sending messages in chunks that are always smaller than the
minimum MTU of all links between the two ends of the socket, you'll
avoid fragementation. But that limit tends to be at an annoying point
that results in test code working but production code failing. :^(
None of this is an issue with blocking sockets (where Tcl can do all the
reassembly for you) or when reading by the line (when the [fblocked]
command is useful if you've got non-blocking sockets in the mix).

Donal.