muse on Compact Structs, pack/unpack

#[ are there people paying attention to these issues on other mailing lists?
 ]

= on Compact structs
revision 1, initial posting

What functions serialize/deserialize to the C view?

If these are to be member functions, they would be applicable only if the st
ruct is compact, and erroneous to call otherwise.  It seems like a compact s
truct ought to have a role supported for that purpose, and to make a compact
 struct you declare that ro
le.  This would then allow the compiler to check at compile-time that all th
e properties were indeed native types, things also supporting the compact ro
le, or declared to be non-state data.

Any class could be forced to be compact regardless of contents if it explici
tly supplied the serialization functions.

There are some cases where a memory layout can be variable, such as a polymo
rphic type.  The top-level class can read just the first few bytes, determin
e the correct variant, and create an instance of that.  The class would hold
 a pointer to the variant,
and not appear to be a compact struct, except for the custom logic implement
ed in the pack/unpack functions.

A structure may be variable length, having a length-prefixed string or other
 array, immediately followed by other fields.  Having a @list of things as a
n attribute would normally render it non-compact, but it could have other pr
operties attached that tell
it how to read/write that list, in the spirit of IDL.  If nothing is availab
le that work in this case, then attaching a custom reader/writer property to
 just that attribute would do the trick, something sourly missing in systems
 such as .NET Serializatio
n.

class example_record
does Compact
is finalized
{
has Str $.name
is rw
should serialize (:length_prefix(2), :encoding(UTF16),
padding => {:char("\0", :strip} );
has int16 $.val is rw;
has Str $!presentation_cache;  # private, not included in compact struct I/O
method presentation () returns Str
{
unless defined $!presentation_cache {
.. # compute it
}
return $!presentation_cache;
}
}

I'm not entirely sure from S12, so I'd like confirmation on this:  The avail
ability of trait 'serialize' is a role nested inside role Compact, so it is 
available (with this meaning) in this scope.

My example uses a mundane int16, which knows to pack/unpack as 2 bytes.  But
 it also has a length-prefixed string, which is explained using the trait, s
o the class as a whole is still able to pack/unpack.  It also has private da
ta, which by default is ign
ored by the pack/unpack logic and does not interfere with its ability.

This is what I envision for a class that can take the place of pack/unpack i
n a declarative way.  I've used pack/unpack for disk-based records and wishe
d that the P5 ability was extensible to my own codes to go with already-defi
ned packable structures.  I
've also used .NET serialization and found its declarative ability to be lac
king to the point where it is often easier to write the function from scratc
h.

Meanwhile, how do I use it?

my Buf $temp = $record;
$stream.print ($temp);

$stream.print (Buf $record);

That is a bit baroque.  Two issues here: is 'print' still the only way to ou
tput?  I think 'print' would convert all arguments to string form, so printi
ng an int16 would format the number into text.  It also deals with encoding 
issues.  Printing a Buf wou
ld seem to turn that Buf back into a Str using all the rules set up for that
.

So how about a binary output function, 'write'?  It will imply binary output
, so knowing this can help hide the differences between text/binary on platf
orms that have it.  It will convert its arguments to Buf by default.

$stream.write ($record);  # just what I need

my int16 $x = 42;
$stream.write ($x);  # emits 2 bytes, exactly as stored in the primitive
$stream.print ($x);  # emits characters "4" and "2" in the proper encoding

So as mentioned at the start of this muse, where do the functions live?  It 
is stated that the type name used as a listop is a conversion function.  Is 
that special syntax that knows to look for some way to accomplish that?  I'l
l proceed on this assumptio
n, and basic C++ ideas as a strawman, that it looks for things in each of th
e two classes (coming and going) and built-in rules, perhaps a chain of thin
gs.

Suppose that one of the places it looks is

multi conversion:<Buf> () { ... }

that can take adjectives to control the conversion, or additional positional
 arguments.

This is supplied by the Compact role, whose implementation handles common ad
verbs and invariant logic.  But it calls another function, pack, for the bas
ic packing work.  The version supplied with the Compact role would know abou
t primitive types and to re
curse on other Compact items, and iterate over the contents of the class.  T
o do something different, a class can write its own pack function with the r
equired signature to knock-out the default version in the role.

class VLI<2.0.1 cpan:DLUGOSZ>
does Compact
{
# implements packing/unpacking as described in <http://www.dlugosz.com/ZIP2/VLI.ht
ml>
has Int $.value is rw;
multi pack (*%adverbs) returns Buf
{
.. respects standard signed/unsigned option,
supports unique options for encoding variations
}
} # end class VLI

That is an example where my "Compact"ness can clearly be seen as a pack/unpa
ck format.  Other examples would be structures that appear in the Win32 API,
 that I can discuss if necessary.

Now something I've learned from doing .NET serialization (XML as it happens,
 but that does not matter to the issue).  Here I have an Int, that is a perf
ectly ordinary Int like any other except for its serialization logic.  I can
 use it in a larger class t
o make it serialize how I want.  But to access the value in that class, I ha
ve an extra layer of indirection.

class C does Compact is rw
{
has int32 $x;
has VLI $y;
}

my C $c;  # for a Compact class, empty prototype is not undef but default va
lues
$c.x= 5;  # OK
$c.y.value = 1234567812345678;  # yuck

I'd really like to attach the serialization semantics class to the Int decla
red in the outer class, where that semantics class does not contain the data
 itself.

class VLI<2.0.1 cpan:DLUGOSZ>
does Compact::helper
{
# implements packing/unpacking as described in <http://www.dlugosz.com/ZIP2/VLI.ht
ml>
multi pack (Int value, *%adverbs) returns Buf
{
.. respects standard signed/unsigned option,
supports unique options for encoding variations
}
} # end class VLI

class C does Compact is rw
{
has int32 $x;
has Int $y is VLI;
}

Finally, different kinds of serialization can use similar mechanisms, and mo
re than one can be applied to the same class and all the traits/properties/r
oles should play nice with each other.  Such combined classes should also ca
scade.

On Apr 1, 2008, at 6:13 , John M. Dlugosz wrote:
> Meanwhile, how do I use it?
>
> 	my Buf $temp = $record;
> 	$stream.print ($temp);
>
> 	$stream.print (Buf $record);

> $stream.print($record.pack) # I would think?
>

--
brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com
system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu
electrical and computer engineering, carnegie mellon university    KF8NH

Brandon S. Allbery KF8NH allbery-at-ece.cmu.edu |Perl 6| wrote: 
>
Thank you so much for the reply.  I was beginning to wonder if Perl 6
interest is dead, or if there is another secret lair for current
enthusiasts that I'm unaware of.

A .pack member function on a Compact struct is indeed my first gut
feeling, but at the end of the section he states specifically that
packing is triggered by coercing  to a Buf type.

Since nothing is sacred, I would propose that .pack is more sensible as
the basic mechanism, to be used when doing this explicitly.  And a
conversion operation to Buf can be provided that simply calls .pack().

--John

On Tue, Apr 01, 2008 at 12:25:50PM -0500, John M. Dlugosz wrote:
> Thank you so much for the reply.  I was beginning to wonder if Perl 6
> interest is dead, or if there is another secret lair for current
> enthusiasts that I'm unaware of.

Sorry, many of us have to live under budgetary constraints of time
and health and energy (not to mention money), and we'll need just
a bit more advance notice if we're to schedule our vacations and/or
manic periods to coincide with your current productive period.  :)

'Course, if you happen to know any billionaires who want to drop a wad
on our sorry society, I'm sure we can all figure out a few ways to use
money as a proxy for other forms of happiness...

Personally, my brain is rather occupied with a Hard Problem at my
$dayjob, not to mention thinking about Longest Token Matching and Taxes
and Family and Church, and all the Speeches I'll have to give in the
coming few months if the Doctors don't lock me up Somewhere Safe...

In general, however, I do eventually get around to thinking about
topics raised on this mailing list.  But I do not have an ADHD brain,
and switching topics is a high-overhead activity for me, so I prefer
to think about things when I have enough time to think about them
thorougly.  Long treatises particularly take a while to digest, so
to the extent that you have been splitting things up into separate
topics, I appreciate it greatly.  It also helps me keep track of
which bits I've responded to and which I haven't yet.

I admit that I also like to sit back and let other people pick the
low fruit so I don't have to spend time doing it.  Over the short
term, sometimes what I do looks an awful lot like doing nothing... :)

Larry

On Apr 1, 2008, at 13:25 , John M. Dlugosz wrote:
> Brandon S. Allbery KF8NH allbery-at-ece.cmu.edu |Perl 6| wrote: 
> A .pack member function on a Compact struct is indeed my first gut
> feeling, but at the end of the section he states specifically that
> packing is triggered by coercing  to a Buf type.
>
> Since nothing is sacred, I would propose that .pack is more
> sensible as the basic mechanism, to be used when doing this
> explicitly.  And a conversion operation to Buf can be provided that
> simply calls .pack().

I can see the point of autocoercion, but (a) that it exists doesn't
mean it's the only way to do it (this *is* Perl...), (b) we already
have other coercions such as .perl and .yaml so this would be
consistent.  (Then again, that implies postfix .str instead of/in
addition to prefix (~).  Hm.)

--
brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com
system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu
electrical and computer engineering, carnegie mellon university    KF8NH

I take that as encouragement, and confirmation that this is the right
place to post (and archive) my musings.  HOPEFULLY I can get questions
answered too.

I credit the success of C++, in part, to Stroustrup's personal training
of the first gurus.  He would explain things that made me see the
underlying principles, and "the way" of C++.  Eventually I was given
early drafts of the 2.0 spec, and I had a talent for spotting
inconsistencies in the technical docs.  My commentary and feedback
contributed to the eventual published ARM, and the first year of the
ANSI process.

Having done that before, I find the Perl 6 technical docs to be in
relative disarray and imprecise.  A sticking point such as the subscript
question really bugs me and I have a hard time moving forward.  An
answer of "not sure; that section needs work" is fine, with or without
"what do you propose?".

I'm on a sabbatical now, and learning Perl 6 is one of my priorities.  I
was hoping to accomplish more during my month in China, but had less
down-time then feared.  But I caught up to the current S\d\d files and
started organizing my wetware.

I have other "musings" I've filed away, thinking it may clear up some as
I integrate the other documents and discussions in the archives, and
because I don't want to overload the discussion board.

So excuse my manic period; posting around the clock is due to jet lag.
I hope my "fresh eyes" and analysis can make a contribution.  But I
don't want to re-invent something that is already done, so please steer
me in the right direction when I'm NOT exploring the frontier but have
merely missed the path.

--John
(literally not sure if it was AM or PM; wife laughed when I asked for
pancakes)


Larry Wall larry-at-wall.org |Perl 6| wrote:
> On Tue, Apr 01, 2008 at 12:25:50PM -0500, John M. Dlugosz wrote:
> 
>
> Sorry, many of us have to live under budgetary constraints of time
> and health and energy (not to mention money), and we'll need just
> a bit more advance notice if we're to schedule our vacations and/or
> manic periods to coincide with your current productive period.  :)
>
> 'Course, if you happen to know any billionaires who want to drop a wad
> on our sorry society, I'm sure we can all figure out a few ways to use
> money as a proxy for other forms of happiness...
>
> Personally, my brain is rather occupied with a Hard Problem at my
> $dayjob, not to mention thinking about Longest Token Matching and Taxes
> and Family and Church, and all the Speeches I'll have to give in the
> coming few months if the Doctors don't lock me up Somewhere Safe...
>
> In general, however, I do eventually get around to thinking about
> topics raised on this mailing list.  But I do not have an ADHD brain,
> and switching topics is a high-overhead activity for me, so I prefer
> to think about things when I have enough time to think about them
> thorougly.  Long treatises particularly take a while to digest, so
> to the extent that you have been splitting things up into separate
> topics, I appreciate it greatly.  It also helps me keep track of
> which bits I've responded to and which I haven't yet.
>
> I admit that I also like to sit back and let other people pick the
> low fruit so I don't have to spend time doing it.  Over the short
> term, sometimes what I do looks an awful lot like doing nothing... :)
>
> Larry
>
>

On Tue, Apr 01, 2008 at 09:27:48PM -0500, John M. Dlugosz wrote:
> Having done that before, I find the Perl 6 technical docs to be in relativ
e
> disarray and imprecise.

Indeed, I welcome all the help I can get on making things more precise.
My own tendency is to emphasize vigor over rigor, so I welcome any
balance on that score.  And please don't be discouraged by any of my
traditional morning grumpiness.  I don't need a jet to get jet lagged.  :)

Larry

Larry Wall larry-at-wall.org |Perl 6| wrote:
> On Tue, Apr 01, 2008 at 09:27:48PM -0500, John M. Dlugosz wrote:
> 
>
> Indeed, I welcome all the help I can get on making things more precise.
> My own tendency is to emphasize vigor over rigor, so I welcome any
> balance on that score.  And please don't be discouraged by any of my
> traditional morning grumpiness.  I don't need a jet to get jet lagged.  :)
>
> Larry
>
>
During ANSI/ISO standardization, they basically took every phrase and
made it more and more exact.  It went from understandable to leagaleze
over a period of years, with sentences growing more and more detail.  I
could still follow it having come to it gradually.

Hypertext could remedy that.

--John

On Wed, Apr 02, 2008 at 07:03:46PM -0500, John M. Dlugosz wrote:
> During ANSI/ISO standardization, they basically took every phrase and made
> it more and more exact.  It went from understandable to leagaleze over a
> period of years, with sentences growing more and more detail.  I could
> still follow it having come to it gradually.
>
> Hypertext could remedy that.

"All problems in computer science can be solved by another level
of indirection."  -- Butler Lampson

"...except the problem of too many levels of indirection."
-- Larry's Amendment to Lampson's Law

But yes, it might be about time for hypertexting it.  All but S03 have
never really undergone a major reorg, and most of them could use it.
Maybe it's time to set up Twiki on my home machine...

Larry

Or you could setup pmwiki as a nod to our perl6 compiler pumpking  :-)

In any case, pmwiki is simpler to setup than twiki.

cheers,

-Scott

On Wed, Apr 2, 2008 at 7:45 PM, Larry Wall <larry@wall.org> wrote:

> On Wed, Apr 02, 2008 at 07:03:46PM -0500, John M. Dlugosz wrote: 
> made 
>
> "All problems in computer science can be solved by another level
> of indirection."  -- Butler Lampson
>
> "...except the problem of too many levels of indirection."
>        -- Larry's Amendment to Lampson's Law
>
> But yes, it might be about time for hypertexting it.  All but S03 have
> never really undergone a major reorg, and most of them could use it.
> Maybe it's time to set up Twiki on my home machine...
>
> Larry
>



--
Jonathan Scott Duff
perlpilot@gmail.com