perpetual compression

YE GADS, I read this stuff and regret that I ever worked on
compression.  (Yes, I mean this!  I had better uses for my time.)  But
I've started using the net more, and wow!, sending just a small
signature file instead of the conventionally compressed contents would
be a big savings.  (I call my compressed files 'signatures' because
they are much like an actual signature (they represent the author yet
themselves are so small.)

It's time to move forward.  So, consistent with my goals and what I've
been asked to do, I have been getting ready to do just that (a demo.)

The program does have a few problems though, as it's configured.  But
it works.  This is the main issue:

a)  The output is organized into a series of blocks, (a small header
begins each output section,) and each block consists of two parts:

a1)  Three bytes (in one configuration -- but in any case the number
of bytes is sent as part of the header.)

a2)  A copy of the input block (read via an 'fread'), except the block
contents have been made quite sparse, and thus the contents can be
compressed.

a2-1)  Each block prologue is always less than 128 bytes.

a2-2)  The sparse block component is always a multiple of 128 bytes.

a3)  The header lacks any sort of 32-bit checksum facility to detect
I/O errors.  Whcih, given that this process is serial ain't a good
idea.  On a machine that is fairly new (the disks were purchased less
than two years ago, I have several 100gb hard-drives with 50 and 75gb
files (.tar files,) and two efforts to compress these large files have
failed because of the rare bit error which doesn't get detected.

I need to detect these errors and reconstruct the buffers correctly.
Without an error detection mechanism this process will never work
reliably with large files.

On 27 Oct 2004 11:07:56 -0700, julesg@shilohyrc.com (Jules Gilbert)
wrote:

>a3)  The header lacks any sort of 32-bit checksum facility to detect
>I/O errors.  Whcih, given that this process is serial ain't a good
>idea.  On a machine that is fairly new (the disks were purchased less
>than two years ago, I have several 100gb hard-drives with 50 and 75gb
>files (.tar files,) and two efforts to compress these large files have
>failed because of the rare bit error which doesn't get detected.
>
>I need to detect these errors and reconstruct the buffers correctly.
>Without an error detection mechanism this process will never work
>reliably with large files.

What are you talking about?  Hard disks would give less than 1bit of
unrecoverable error if you ran them 24/7 for nearly 6 months!
If you must detect corruption, just 'waste' 32 bits and use a CRC.

Jim

julesg@shilohyrc.com (Jules Gilbert) :

> YE GADS, I read this stuff and regret that I ever worked on
> compression.  (Yes, I mean this!  I had better uses for my time.)

Now there is a woopping lie, after nine years of promises we still have not
seen a working program of any sort from you.

> But
> I've started using the net more, and wow!, sending just a small
> signature file instead of the conventionally compressed contents would
> be a big savings.  (I call my compressed files 'signatures' because
> they are much like an actual signature (they represent the author yet
> themselves are so small.)

Big deal, people are transfering multi-megabyte media files on the internet
and you worry how much bandwidth signatures take.  What happened to the clai
m
of packing entire videos to a floppy disk?

> It's time to move forward.  So, consistent with my goals and what I've
> been asked to do, I have been getting ready to do just that (a demo.)

I believe it when I see it since you never delivered anything else so far,
otherwise please shut up!

> The program does have a few problems though, as it's configured.  But
> it works.  This is the main issue:

Jee, does this sound like the same old JC garbage,?  "I have this great
program, but it just needs a little fix first."

> a)  The output is organized into a series of blocks, (a small header
> begins each output section,) and each block consists of two parts:
>
> a1)  Three bytes (in one configuration -- but in any case the number
> of bytes is sent as part of the header.)
>
> a2)  A copy of the input block (read via an 'fread'), except the block
> contents have been made quite sparse, and thus the contents can be
> compressed.

And what use is this over a human readable signature?

> a2-1)  Each block prologue is always less than 128 bytes.
>
> a2-2)  The sparse block component is always a multiple of 128 bytes.

And what use is this over a human readable signature?

> a3)  The header lacks any sort of 32-bit checksum facility to detect
> I/O errors.  Whcih, given that this process is serial ain't a good
> idea.  On a machine that is fairly new (the disks were purchased less
> than two years ago, I have several 100gb hard-drives with 50 and 75gb
> files (.tar files,) and two efforts to compress these large files have
> failed because of the rare bit error which doesn't get detected.

Yeah, like I believe this, hundreds of compression programs released all ove
r
the world, and the only ones that fail because a rare disk read errors are
yours.  I think the easier answer is that you never had a working program an
d
now you claim that the reason is that your hard drives have bit errors.

> I need to detect these errors and reconstruct the buffers correctly.
> Without an error detection mechanism this process will never work
> reliably with large files.

Nice con, why not use Google to find some free error correcting code?  My
guess is because you are running out of excuses for admitting you were wrong
from the start.

Earl Colby Pottinger


--
I make public email sent to me!  Hydrogen Peroxide Rockets, OpenBeos,
SerialTransfer 3.0, RAMDISK, BoatBuilding, DIY TabletPC.  What happened to
the time? http://webhome.idirect.com/~earlcp

Earl Colby Pottinger <earlcp@idirect.com> writes:

> julesg@shilohyrc.com (Jules Gilbert) :
> 
>
> Now there is a woopping lie,

What, that he worked on /compression/?

Phil
--
They no longer do my traditional winks tournament lunch - liver and bacon.
It's just what you need during a winks tournament lunchtime to replace lost
... liver.   -- Anthony Horton, 2004/08/27 at the Cambridge 'Long Vac.'

"Jim" <spam@ihug.com.au> wrote in message
 news:i0e0o01vl37rl1hu8ipbe7hglojrbrgrk4@
4ax.com...
> On 27 Oct 2004 11:07:56 -0700, julesg@shilohyrc.com (Jules Gilbert)
> wrote:
> 
>
> What are you talking about?  Hard disks would give less than 1bit of
> unrecoverable error if you ran them 24/7 for nearly 6 months!
> If you must detect corruption, just 'waste' 32 bits and use a CRC.

The dog is always eating Jules' homework.
This is just another example.