Compression - zlib and deque

Author

zlib and deque

J. Campbell

2004-10-04, 3:55 pm

Hello people...first post here.

I have written a c++ program that uses in-memory compression.
Currently, I handle dynamic memory allocation using a custom container
class that essentially performs memory operations for arrays. To use
zlib, I need to have all the data in memory *plus* reserved memory to
hold the zlib output (either compression or uncompression). This is
typically not a problem...however, when the amount of data to be
compressed or uncompressed approaches a significant fraction of the
physical RAM, the zlib operations take an enormous performance hit due
to the overhead of using swap files.

Part of the problem is that (at least the way I'm using it), zlib
requires twice (well actuall a hair more than twice) the memory it
actually needs when compressing (source + dest). I was thinking that
if I could put the data that needs compressing into a deque and then
compress into another deque there would likely be substantial
performance increase, and the need to use swap files wouldn't come
into play until much larger amounts of memory were needed.

I envision compressing data by pop_front from the source deque, and
push_back to destination deque.

This would free up the source deque's memory as it is no-longer
needed, and make it available to destination deque.

Currently, I'm using zlib in a very simple way...I've created member
functions of my container class compressIt() and decompressIt() that
call the zlib functions and then assign the destination buffer to the
container itself. Since I'm sending the zlib functions pointer and
data-length info, I don't see a way to do the same using deque while
removing data from the deque as it's used.

Can anyone point me in the right direction??

Thanks

Joe

Matt Mahoney

2004-10-04, 3:55 pm

"J. Campbell" <mango_maniac@yahoo.com> wrote in message
news:b97c86e1.0410040708.5c3a9a7b@posting.google.com...
> Hello people...first post here.
>
> I have written a c++ program that uses in-memory compression.
> Currently, I handle dynamic memory allocation using a custom container
> class that essentially performs memory operations for arrays. To use
> zlib, I need to have all the data in memory *plus* reserved memory to
> hold the zlib output (either compression or uncompression). This is
> typically not a problem...however, when the amount of data to be
> compressed or uncompressed approaches a significant fraction of the
> physical RAM, the zlib operations take an enormous performance hit due
> to the overhead of using swap files.

The input and output buffers for zlib can be any size, as small as one byte.
You don't need to read all of your data before you start
compressing/uncompressing it.

-- Matt Mahoney

Sponsored Links