| Kyle Sallee 2004-09-28, 8:55 am |
| Download many source tarballs?
Sorcerer system administrators do.
However they download only a small fraction of the
amount of bytes of full bzip2 compressed sources.
Sorcerer used xdelta technology to make source
upgrade patches for updating installed software.
I realized xdelta was not tuned for
generation of deltas on source tarballs.
After 2 w s of design and implementation I finished
creating a delta compressor that generates delta files
that can be bzip2 compressed significantly smaller
than commensurate deltas created by xdelta.
You can download sdelta from http://sdelta.berlios.de
The current application of sdelta within Sorcerer
is not merely for source upgrade patches.
Initial source downloads receive a sdelta patch for
the desired source generated using a generic dictionary.
This has reduced initial transfers of sources by as much
as 46% that I have seen demonstrated thus far.
376356 polypaudio-0.4.tar.bz2
170028 d1-polypaudio-0.4.tar.sdelta.bz2
While I have not yet determined the average savings percent
a preliminary glance and comparisons between sdeltas patches
and bzip2 source tarballs has revelaed sdelta patches
generated against a generic dictionary file have
significantly reduced the size of first time transfers.
Even on poor matches rarely is the sdelta patch file
using a generic dictionary greater in size than the
original source. When that happens the difference
is usually less than a kilobyte.
Application of sdelta technology within the Sorcerer distribution
of GNU Linux and free software has made it the #1 low bandwidth
distribution for advanced administration.
If you have a use for xdelta then you may
want to try sdelta to see how it compares.
sdelta has been tested for creating deltas
for sources tarballs as large as linux.
2053118 linux-2.6.7-linux-2.6.8.1.tar.sdelta.bz2
3190401 linux-2.6.7-linux-2.6.8.1.tar.xdelta.bz2
While both are excellent deltas a matter of one megabyte
or roughly 1/3 better delta generated by sdelta should
make it worth the same consideration that bzip2 gained
by creating smaller archives than gzip.
And for one last example a small file upgrade patch:
3.0M polypaudio-0.4.tar
33759 polypaudio-0.3-0.4.tar.sdelta.bz2
45088 polypaudio-0.3-0.4.tar.xdelta.bz2
The xdelta patch was created using xdelta version 1.1.3
without compression and then bzip2 compressed.
The above examples may not represent the best or the
worst favorable examples towards sdelta technology.
They were what I used for testing sdelta.
By the time you read this more than half a gigabyte
of sdelta source upgrade patches have been generated
on demand and downloaded by Sorcerer boxes.
You can check them out at http://sorcerer.wox.org/download/sdelta
Please carbon copy replies and questions to my email,
because I am not a regular on this list.
I researched and learned compression
technology algorithms less than 3 ws ago.
Please forgive if I if miss, mess up,
or misconstrue the jargon in replies.
|