Compression - non-solid archive format that allow streamed updates?

Author

non-solid archive format that allow streamed updates?

NOBODY

2006-02-25, 6:55 pm

Hi,

Do you know of an 'incremental' archive format that would be suited for
an outputstream?

In other words, is there any archive format that can hold open the
existing entries and allow to append them in an interlaced fashion? (an
incrementally updating archive?)

Let me explain.

Let's say I have 2 types (A nd B) of csv data to send.
A1.csv, A2.csv, A3.csv
B1.csv, B2.csv, B3.csv

I want to write to a stream an archive format that will contain 2
entries (A and B) where A is the contatenation of A1+A2+A3, and B is the
contatenation of B1+B2+B3.

Now, imagine a zip file. It is easy enough to create a new zip entry A,
and push all A1, A2, A3 files in sequence, and create a second zip entry
B and push B1, B2, B3.

But here is the problem: the sequence is rolling (like a log4j
file-size- rolling appender, or logrotate for the non-java folks) and by
the time I finished pushing A3, B1 be has rolled off. I want to push A1
B1, A2 B2, A3 B3.

So, I cannot use zipfile, at least not that I know of, to "append
existing entry" instead of always creating a new entry.

Something smart like gzip (where you can concatenate independant gzip
files and they become a valid single gzip file) only for multiple
entries (that gzip doesn't have) would be great!

Thanks.

Jasen Betts

2006-02-26, 3:55 am

On 2006-02-25, NOBODY <antispam@0.0.0.0> wrote:
> Hi,
>
> Do you know of an 'incremental' archive format that would be suited for
> an outputstream?
>
> In other words, is there any archive format that can hold open the
> existing entries and allow to append them in an interlaced fashion? (an
> incrementally updating archive?)

..arj

> So, I cannot use zipfile, at least not that I know of, to "append
> existing entry" instead of always creating a new entry.
>
> Something smart like gzip (where you can concatenate independant gzip
> files and they become a valid single gzip file) only for multiple
> entries (that gzip doesn't have) would be great!

shar (or shar.gz if you want compression)

shar is basically a shell script...

----EXAMPLE---8<--cut here---
#!/bin/sh
cat > A.csv <<EOF
1,2,3
4,5,6
EOF
cat > B.csv <<EOF
a,b,c
d,e,f
EOF
cat >> A.csv <<EOF
7,8,9
EOF
cat >> B.csv <<EOF
g,h,i
EOF
------>8---cut here-----

If you've got "shar tools" (or a bourne compatible shell) you shoulde be able
to extract the two interlaced files from that. (simtel.net has a version for
dos)

Bye.
Jasen

Sponsored Links