Tcl - Binary data INTO C extension?

Author

Binary data INTO C extension?

2006-03-24, 10:01 pm

Ok, I've spent 2+ hours searching the wiki, google, reading docs, etc..

I think I understand the problem, but don't know the best solution..

I have a C extension (wrapped via SWIG) that needs to get access to some
binary data read in by my tcl code.

I've done something like this:
set fp [open 1.png r]
fconfigure -translation binary $fp
set fdat [read $fp]
close $fp

Now, if I do this:
set fp [open 2.png w]
fconfigure -translation binary $fp
puts -nonewline $fp $fdat
close $fp

then I get an exact copy of my file (i.e. Tcl is self-consistent).

On the other hand, if I call my swig-wrapped extension,
addImage $fdat [string length $fdat]

(prototype for addImage in swig's .i file is: void addImage(char *data, int
len);)

I get what I believe is a UTF-8 encoded version of my data (for instance,
using od -a 1.png, what is a 'nul' character is expanded to 2 characters '@
nul', etc). This is somewhat supported by the fact that [string bytelength
$fdat] is always longer than [string length $fdat].

How do I get the data to be simply in regular char * format? If I do an
fread of this data, or even if I do the open from Tcl, pass the channel name
and do Tcl_GetChannel/Tcl_Read, it works just fine.

Do I have to mess with typemaps in swig?

Can this even be done? I've tried various "encoding convertto" operations,
and I think even a few "binary format" ops, and nothing seems to work. They
DO produce interesting results, but nothing directly..

Comments?

Thanks,
Mattias

Michael Schlenker

2006-03-24, 10:01 pm

MH schrieb:
> Ok, I've spent 2+ hours searching the wiki, google, reading docs, etc..
>
> I think I understand the problem, but don't know the best solution..
>
> I have a C extension (wrapped via SWIG) that needs to get access to some
> binary data read in by my tcl code.
>
> I've done something like this:
> set fp [open 1.png r]
> fconfigure -translation binary $fp
> set fdat [read $fp]
> close $fp
>
> Now, if I do this:
> set fp [open 2.png w]
> fconfigure -translation binary $fp
> puts -nonewline $fp $fdat
> close $fp
>
> then I get an exact copy of my file (i.e. Tcl is self-consistent).
>
> On the other hand, if I call my swig-wrapped extension,
> addImage $fdat [string length $fdat]
>
> (prototype for addImage in swig's .i file is: void addImage(char *data, int
> len);)
>
> I get what I believe is a UTF-8 encoded version of my data (for instance,
> using od -a 1.png, what is a 'nul' character is expanded to 2 characters '@
> nul', etc). This is somewhat supported by the fact that [string bytelength
> $fdat] is always longer than [string length $fdat].
>
> How do I get the data to be simply in regular char * format? If I do an
> fread of this data, or even if I do the open from Tcl, pass the channel name
> and do Tcl_GetChannel/Tcl_Read, it works just fine.
>
> Do I have to mess with typemaps in swig?
>
> Can this even be done? I've tried various "encoding convertto" operations,
> and I think even a few "binary format" ops, and nothing seems to work. They
> DO produce interesting results, but nothing directly..
>
> Comments?
>
Don't know enough about swig, but what you need is
Tcl_GetByteArrayFromObj()...

Michael

Don Porter

2006-03-24, 10:01 pm

MH wrote:
> On the other hand, if I call my swig-wrapped extension,
> addImage $fdat [string length $fdat]
>
> (prototype for addImage in swig's .i file is: void addImage(char *data, int
> len);)
>
> I get what I believe is a UTF-8 encoded version of my data ...

The Tcl_ObjCmdProc command procedure for your [addImage] command
needs to use Tcl_GetByteArrayFromObj() to retrieve the bytes from
the argument values passed in.

I don't know SWIG, so I don't know what hoops, if any, you can
jump through to make it do this for you.

--
| Don Porter Mathematical and Computational Sciences Division |
| donald.porter@nist.gov Information Technology Laboratory |
| http://math.nist.gov/~DPorter/ NIST |
|_______________________________________
_______________________________|

Paul Obermeier

2006-03-24, 10:01 pm

Hello Matthias,

What you want, can be done.
But, you have to mess with SWIG's typemaps (a little bit).

You may take a look at my Tcl3D extension at www.tcl3d.org, where I do
similar things.

File tcl3dUtil/swigfile/bytearray.i defines typemaps and utility functions
to copy data from a Tcl binary string into the format needed by OpenGL.
In your case, it may not be necessary to do the copying step done in
function tcl3dByteArray2Vector.

Greetings,

Paul

"MH" <mghembru@harshrealm.uwaterloo.ca> schrieb im Newsbeitrag
news:e01h9l$ndg$1@rumours.uwaterloo.ca...
> Ok, I've spent 2+ hours searching the wiki, google, reading docs, etc..
>
> I think I understand the problem, but don't know the best solution..
>
> I have a C extension (wrapped via SWIG) that needs to get access to some
> binary data read in by my tcl code.
>
> I've done something like this:
> set fp [open 1.png r]
> fconfigure -translation binary $fp
> set fdat [read $fp]
> close $fp
>
> Now, if I do this:
> set fp [open 2.png w]
> fconfigure -translation binary $fp
> puts -nonewline $fp $fdat
> close $fp
>
> then I get an exact copy of my file (i.e. Tcl is self-consistent).
>
> On the other hand, if I call my swig-wrapped extension,
> addImage $fdat [string length $fdat]
>
> (prototype for addImage in swig's .i file is: void addImage(char *data,
int
> len);)
>
> I get what I believe is a UTF-8 encoded version of my data (for instance,
> using od -a 1.png, what is a 'nul' character is expanded to 2 characters
'@
> nul', etc). This is somewhat supported by the fact that [string bytelength
> $fdat] is always longer than [string length $fdat].
>
> How do I get the data to be simply in regular char * format? If I do an
> fread of this data, or even if I do the open from Tcl, pass the channel
name
> and do Tcl_GetChannel/Tcl_Read, it works just fine.
>
> Do I have to mess with typemaps in swig?
>
> Can this even be done? I've tried various "encoding convertto" operations,
> and I think even a few "binary format" ops, and nothing seems to work.
They
> DO produce interesting results, but nothing directly..
>
> Comments?
>
> Thanks,
> Mattias

Kip Rugger

2006-03-25, 7:03 pm

In article <e01h9l$ndg$1@rumours.uwaterloo.ca>,
MH <mghembru@harshrealm.uwaterloo.ca> wrote:
[snip]
>
>Do I have to mess with typemaps in swig?
>

The SWIG typemap is actually quite simple. Here is a sample
SWIG file, which I will call "bytearray.i":

----
%module MODULE

%typemap(in) char* bytearray
"$1 = Tcl_GetByteArrayFromObj($input, 0);";

// the following is a swig wrapper def, not a C prototype
// note that the parameter name must be "bytearray" in order to invoke the
// typemap just defined
int sample_function(char* bytearray, int);

%{ // this is inlined C code, which is copied verbatim to the output
// in a real project it would likely be in another (.c) file

// this is a C prototype, not processed by SWIG
int sample_function(char* arr, int len);

// this is a C function, not processed by SWIG
int
sample_function(char* arr, int len)
{
return 17;
}

%}
----

and here is a somewhat generic Makefile to compile it:
----
MODULE = bytearray
SRC =
SWIGSRC = $(MODULE).i
OBJ = $(SRC:.c=.o) $(SWIGSRC:.i=.o)
TARGET = lib$(MODULE).so

CFLAGS = -O2 -fpic
CPPFLAGS = -I/usr/local/tcltk/include -DUSE_TCL_STUBS

SWIGDIR = /usr/local/swig
SWIG = $(SWIGDIR)/bin/swig

TCLLIB = /usr/local/tcltk/lib

LIBS = -L$(SWIGDIR)/lib -lswigtcl8 -L$(TCLLIB) -ltclstub8.4

%.c:%.i
$(SWIG) -c -tcl -DMODULE=$(MODULE) -o $@ $?

$(TARGET): $(OBJ)
$(CC) -shared -o $@ $? $(LIBS) -Wl,-rpath,$(SWIGDIR)/lib

clean::
rm -f $(OBJ) $(SWIGSRC:.i=.c) $(TARGET)
----

After building the shared library, you can use your new extension thusly:
----
tcl> load ./libbytearray.so
tcl> sample_function abc 3
17

2006-03-27, 7:04 pm

In article <e0458u$n1$1@darkstar.ca>, Kip Rugger <kr@nodomain.net> wrote:
>In article <e01h9l$ndg$1@rumours.uwaterloo.ca>,
>MH <mghembru@harshrealm.uwaterloo.ca> wrote:
>[snip]
>
>The SWIG typemap is actually quite simple. Here is a sample
>SWIG file, which I will call "bytearray.i":

[cut]

Thanks, Paul and Kip.. I will play with it a bit later today..

Am trying to solve a different problem first :-)

Mattias

Sponsored Links