Smalltalk - Sending floats over a client-server in Smalltalk

Author

Sending floats over a client-server in Smalltalk

enigmainorlando@aol.com

2006-04-05, 4:07 am

Hi ya,

This is sorta related to Smalltalk, sorta related to client-server, so
I hope it's okay to post here. If it isn't, I don't mind going
elsewhere. But anyway, here goes.

I'm currently breaking my application up into a client and a server.
I've never done a client-server before, so this is the first time. I'm
using TCP for the time being, and VisualWorks 7.3. I'm no guru, and am
just figuring out things as I go. I don't even have a book (but then
again, I've never have books - I just "do it").

Anyway the server needs to send floating point numbers (an array of
them, actually) that are computed on the fly, to the client upon
demand. I've already got characters flying back and forth, as a test
case. But apparently (again, this is the first time I'm doing this -
I'm no client-server guru), characters (or "character-like" data)
appear to be only thing that is supported as far as sending data over
TCP connections. Am I right about this?

I already understand that I could simply convert each floating point
digit to a char and send them that way. But that seems to be
ineffecient, because I tested it out by timing the conversion from
float to string, and efficiency in this client-server will be priority,
as this stuff will be real-time. The other scheme I came up with, off
the top of my head (again, I'm no guru, and am just figuring things out
as I go) was to get rid of the "float" part of the floating point by
multiplying all floats by 10000, so that 1.23456 becomes 123456. This
conversion, to my surprise, was several hundred times faster in terms
of performance. My thoughts were that I could then send this number
over through binary mode vs. character mode. But the number seems too
big - I believe that I can only send over numbers that are
"character-sized" for lack of better words.

Any answers to the following would be appreciated.

1) What is the "standard" way of sending floats over a tcp connection,
keeping performance in mind?

2) Are there any VW specific (or Smalltalk-specific) ways of doing
this?

3) Can one send rather large interger data (say 123456) over a tcp
connection? Is there some way to "encode" this, or break it up into
smaller bytes, or whatever?

4) Is the "multiply the floats by 10000" approach a good approach, or
is this laughable because there is some much better way to do it?

Thanks.

Marten Feldtmann

2006-04-05, 8:03 am

enigmainorlando@aol.com schrieb:

>
> Any answers to the following would be appreciated.
>
> 1) What is the "standard" way of sending floats over a tcp connection,
> keeping performance in mind?
>
> 2) Are there any VW specific (or Smalltalk-specific) ways of doing
> this?
>
> 3) Can one send rather large interger data (say 123456) over a tcp
> connection? Is there some way to "encode" this, or break it up into
> smaller bytes, or whatever?
>
> 4) Is the "multiply the floats by 10000" approach a good approach, or
> is this laughable because there is some much better way to do it?
>
> Thanks.
>

I would guess, that the conversion of a float to a string (printString)
consumes less time than sending the stuff via net - therefore convert
the numbers to strings and send strings (remember the different NLS
requirements for the decimal point (decimal comma) on each side).

If your stuff works (with strings), you may change the transfer
protocol and send 4/8 binary bytes (IEEE format) of your float/double
over the net - but be aware of different CPU archtecture (little/big
endian).

But if I remember correct there is also a network binary
representation, which is independent of the CPU architecture and I also
think, that VW supports this.

Marten

--
Marten Feldtmann - Germany - Software Development
Information regarding VA Smalltalk and DMS-system
"MSK - Mien Schrievkrom" at: www.schrievkrom.de

David Buck

2006-04-05, 8:03 am

Send the bytes of the float and reconstruct them at the other end.

| value byte1 byte2 byte3 byte4 newValue |
value := 5.6.

"Split the number into bytes."
byte1 := value basicAt: 1.
byte2 := value basicAt: 2.
byte3 := value basicAt: 3.
byte4 := value basicAt: 4.

"Reconstruct the bytes"
newValue := Float basicNew: 4.
newValue basicAt: 1 put: byte1.
newValue basicAt: 2 put: byte2.
newValue basicAt: 3 put: byte3.
newValue basicAt: 4 put: byte4.
newValue

David Buck

enigmainorlando@aol.com wrote:
> Hi ya,
>
> This is sorta related to Smalltalk, sorta related to client-server, so
> I hope it's okay to post here. If it isn't, I don't mind going
> elsewhere. But anyway, here goes.
>
> I'm currently breaking my application up into a client and a server.
> I've never done a client-server before, so this is the first time. I'm
> using TCP for the time being, and VisualWorks 7.3. I'm no guru, and am
> just figuring out things as I go. I don't even have a book (but then
> again, I've never have books - I just "do it").
>
> Anyway the server needs to send floating point numbers (an array of
> them, actually) that are computed on the fly, to the client upon
> demand. I've already got characters flying back and forth, as a test
> case. But apparently (again, this is the first time I'm doing this -
> I'm no client-server guru), characters (or "character-like" data)
> appear to be only thing that is supported as far as sending data over
> TCP connections. Am I right about this?
>
> I already understand that I could simply convert each floating point
> digit to a char and send them that way. But that seems to be
> ineffecient, because I tested it out by timing the conversion from
> float to string, and efficiency in this client-server will be priority,
> as this stuff will be real-time. The other scheme I came up with, off
> the top of my head (again, I'm no guru, and am just figuring things out
> as I go) was to get rid of the "float" part of the floating point by
> multiplying all floats by 10000, so that 1.23456 becomes 123456. This
> conversion, to my surprise, was several hundred times faster in terms
> of performance. My thoughts were that I could then send this number
> over through binary mode vs. character mode. But the number seems too
> big - I believe that I can only send over numbers that are
> "character-sized" for lack of better words.
>
> Any answers to the following would be appreciated.
>
> 1) What is the "standard" way of sending floats over a tcp connection,
> keeping performance in mind?
>
> 2) Are there any VW specific (or Smalltalk-specific) ways of doing
> this?
>
> 3) Can one send rather large interger data (say 123456) over a tcp
> connection? Is there some way to "encode" this, or break it up into
> smaller bytes, or whatever?
>
> 4) Is the "multiply the floats by 10000" approach a good approach, or
> is this laughable because there is some much better way to do it?
>
> Thanks.
>

Mike Hales

2006-04-05, 10:01 pm

I did exactly this kind of thing to send arrays of different kinds of
numbers encoded as strings. The trick is knowing what to decode them
to. What I did was to encode the type of number in the first byte of
the string. Then encode the number in the remaining bytes. That way I
could encoded abitrary numbers and decode them without having to know
what they were.

For integers I implemented the following

Integer>>specialEncoding

| size value |
size := self digitLength + 1.
value := ByteString new: ( size ).
value at: 1 put: self code."the code for integers, should have one for
positive and another for negative values"
2 to: size do: [ :idx | value basicAt: idx put: ( self digitAt: idx -
1 ) ].
^value

To decode read the first byte to determine the type and decode
accordingly

For positive integers

| value |
value := LargePositiveInteger basicNew: self size - 1.
2 to: self size do: [ :idx | value digitAt: idx - 1 put: ( self
basicAt: idx ) ].
^value compressed "the compressed call will return a SmallInteger if it
can"

For negative integers just decode like a positive and multiply by -1

For Floats and Doubles you can take advantage of a couple primitives in
ByteArray. You can just re-implement in ByteString and call the same
primitive.

ByteString>>floatAt:put:
"Store a single-precision (32 bit)
floating point quantity
starting at the given byteIndex."

<primitive: 549>
^self primitiveFailed

ByteString>>floatAt: byteIndex
"Answer a single-precision (32 bit) floating point quantity
starting at the given byteIndex."

<primitive: 548>
^self primitiveFailed

Then to encode implement

Float>>specialEncoding
| value |
value := ByteString new: 5.
value at: 1 put: self code."the code for floats"
value floatAt: 2 put: self.
^value

There are analogous primitives for Doubles

ByteString>>doubleAt: byteIndex put: aDouble
"Store a double-precision (64 bit) floating point quantity
starting at the given byteIndex."

<primitive: 551>
^self primitiveFailed

ByteString>>doubleAt: byteIndex
"Answer a double-precision (64 bit) floating point quantity
starting at the given byteIndex."

<primitive: 550>
^self primitiveFailed

Then to encode implement

Double>>specialEncoding
| value |
value := ByteString new: 9.
value at: 1 put: self code."the code for doubles"
value doubleAt: 2 put: self.
^value

You just have to come up with a few codes to identify your number
types. I did this for Strings and Symbols too so I could polymorphicly
encode most of the "primitive data types" (sounds horrible to
smalltalkers) by sending specialEncoding. I could then take a string
and send it decode and it would look at the first byte, figure out what
it had inside and decode it properly.

Then to do this with collections I would put whole collections into one
big string (for sockets you could simply stream right onto them). I
would encode the length of the first encoded string in the first byte
of the big string then copy the small string in. Then put the next
length and the next string etc. To decode read the first byte for the
length, copy that many bytes then ready the next length.

I hope this makes sense and helps.

Mike

enigmainorlando@aol.com

2006-04-06, 4:06 am

David Buck - thanks, this actually looks exactly like what I'm looking
for. If it works, that is (had no idea about "basicAt" giving access
to the underlying bytes of the float, but forgive me because Smalltalk
is so huge it's hard to know everything).

To everyone else offering suggestions, thanks as well, and I will
continue to monitor responses.

Thanks.

Reinout Heeck

2006-04-06, 4:06 am

enigmainorlando@aol.com wrote:
[...]
>
> Anyway the server needs to send floating point numbers (an array of
> them, actually) that are computed on the fly, to the client upon
> demand. I've already got characters flying back and forth, as a test
> case. But apparently (again, this is the first time I'm doing this -
> I'm no client-server guru), characters (or "character-like" data)
> appear to be only thing that is supported as far as sending data over
> TCP connections. Am I right about this?
[...]

Either characters or bytes depending on whether you configured your
socket's stream as #binary or not.

The process of transforming complex data to a stream of bytes and back
agian is called marshalling/unmarshalling or serializing/deserializing.

For VisualWorks there are various libraries available that already do
that, a obvious one being BOSS, but you can also look at the marshallers
in the OpenTalk St-St framework.

Another interesting candidate is SRP (State replication protocol) since
it is implemented on several St implementations meaning that you can
choose different dialects for your client and server.

SRP is a CampSmalltalk project and can be found here:

http://wiki.cs.uiuc.edu/CampSmallta...ect+interchange

HTH,

Reinout
-------

Thomas Gagne

2006-04-06, 7:04 pm

Reinout Heeck wrote:
> enigmainorlando@aol.com wrote:
> [...]
> [...]
>
> Either characters or bytes depending on whether you configured your
> socket's stream as #binary or not.
>
>
>
> The process of transforming complex data to a stream of bytes and back
> agian is called marshalling/unmarshalling or serializing/deserializing.
>
Is your client another ST application? Then serializing might work
wonderfully.

Another approach might be to use someone else's middleware to manage the
connection between your client and server. TCP communications,
depending on how complex your dialogs, can be difficult. Clients crash,
servers crash, networks crash, do you know how to handle all these
things? Do you want do know how to handle them? Is knowing this socket
stuff core to your application's purpose or is it a distraction?

If the other end of the pipe isn't Smalltalk and might live on a CPU
with different byte ordering it's important to make sure you write
binary data in a specific order (as the example did with #basicAt:) and
read it back in the same order. In the TCP information you've read
already that was the purpose of the C-language macros htonl, htons,
ntohl, and ntohs. Floating point data is yet another problem because
different architectures rrepresent floating points in completely
different formats.

Everything I've done (except my middleware) was deliberate translated
into ASCII so there was no problem marshalling and unmarshalling, and
snooping the traffic was easy (it was, in fact, a goal), and debugging
was easy as well. This is another reason why subsequent distributed
processing projects I've worked on now use XML to format the data
traveling to and fro. I can make sure the request is properly formatted
before the server even tries to see what's inside.

enigmainorlando@aol.com

2006-04-07, 4:06 am

Thanks, Mike Hales. I already have something basic and simple going,
using David Buck's suggestions. I didn't know about the primitives in
BinaryStorageBytes, which I just checked out after seeing your post. I
think I'll reimplement what I have now with the primitives you pointed
out and see if the performance is better, which I presume it should be
since these are specialized primitives.

To Reinout Heeck, thanks for the suggestions. I sorta looked into BOSS
and what not when I first started, but it didn't seem intuitive, and
there were no docs handy on how to use any of that stuff, so I skipped
it. I might check into it more.

Thomas Gagne, yes the clients and the server are all Smalltalk --
VisualWorks in fact. Hadn't thought of using someone else's
middleware. You are right in that I could really care less about all
this client-server stuff, and yes it's a distraction from what is
really going on. I suppose I could check into this avenue if things
end up getting more complicated or complex down the road.

I worried about byte ordering and what not earlier on, but then figured
it might not be a big deal, because I figured I am using VisualWorks on
both ends, and I figured a float in VisualWorks would be ordered the
same way no matter what system it was running on. But, alas, I'm sure
I have figured wrong on this, so when it ends up biting me later on,
I'll look into it.

Thanks for the posts. A big help already.

Martin Kobetic

2006-04-07, 7:04 pm

enigmainorlando@aol.com wrote:
> Thomas Gagne, yes the clients and the server are all Smalltalk --
> VisualWorks in fact. Hadn't thought of using someone else's
> middleware. You are right in that I could really care less about all
> this client-server stuff, and yes it's a distraction from what is
> really going on. I suppose I could check into this avenue if things
> end up getting more complicated or complex down the road.

In that case you probably should check out the doc/OpentalkDevGuide.pdf.
Opentalk-STST is a transparent remote messaging middleware which makes
remote communications over TCP/IP about as easy as it gets.

And yes, the layout of Float does follow the native plaform layout to
avoid costly translations for FP operations, so you would have to do
some byte swapping between platforms with different endianness.
Opentalk-STST of course takes care of that too.

HTH,

Martin

Sponsored Links