Home > Archive > Compression > August 2005 > Zlib: how to have a header and compressed data
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Zlib: how to have a header and compressed data
|
|
|
| Hello
I have some data that I can compress into a file with Zlib.
Now, I'd like to have a header for this file that describes the data.
Since the header information depends on the data, it means the header
will need to be written in the file later on.
How is it possible to write this block of data at the begin of the file
with zlib ?
Paul D.
| |
| jackokring@yahoo.com 2005-07-26, 8:59 am |
| you probably need to cat the data on the front later.
| |
|
| And how you do that with zlib ? (without erasing all the data)
PAul D.
| |
|
|
|
| Hello Mark,
Thks for tip
I tried couple things with gzdopen but I always ended up with all the
previous data erased when I write the header block.
Any ideas ?
Thks
Paul D.
| |
| John Reiser 2005-07-27, 5:01 pm |
| > I tried couple things with gzdopen but I always ended up with all the
> previous data erased when I write the header block.
Create the smallest possible example which fails: 25 lines of code
should be more than enough. Post the code; include name and version
of the environment (operating system, compiler, code libraries).
--
| |
|
| Hi
I am using VC6 under XP Pro, with latest zlib version (1.2.3)
here is piece of code that should write data into a zlib file, then it
should write at the beginning of this file some header information.
(but it fails)
void writeData()
{
// open the file for writing data
FILE *f = fopen(path.c_str(),"wb");
if(!f)
return false;
int fd = fileno(f);
zlibDataFile = gzdopen(fd, "wb6");
// let some space for the header data to be written later on
gzs (zlibDataFile, sizeof(HeaderData), SEEK_SET);
// now write the data
int err = gzwrite(zlibDataFile, .....);
......
gzclose(zlibDataFile);
zlibDataFile = 0;
fclose(f);
}
void writeHeaderData()
{
// open the file for writing header data at the top of the zlib file
FILE *f = fopen(path.c_str(),"wb");
if(!f)
return false;
int fd = fileno(f);
// set the headerData
HeaderData myHeaderData;
zlibDataFile = gzdopen(fd, "wb6");
// write the header data
//******* when I do that all previous data are erased !!!!
int err = gzwrite(zlibDataFile, (char*) &myHeaderData,
sizeof(HeaderData));
gzclose(zlibDataFile);
zlibDataFile = 0;
fclose(f);
}
Regards,
Lionel
| |
|
| Hi
I am using VC6 under XP Pro, with latest zlib version (1.2.3)
here is piece of code that should write data into a zlib file, then it
should write at the beginning of this file some header information.
(but it fails)
void writeData()
{
// open the file for writing data
FILE *f = fopen(path.c_str(),"wb");
if(!f)
return false;
int fd = fileno(f);
zlibDataFile = gzdopen(fd, "wb6");
// let some space for the header data to be written later on
gzs(zlibDataFile, sizeof(HeaderData), SEEK_SET);
// now write the data
int err = gzwrite(zlibDataFile, .....);
......
gzclose(zlibDataFile);
zlibDataFile = 0;
fclose(f);
}
void writeHeaderData()
{
// open the file for writing header data at the top of the zlib
file
FILE *f = fopen(path.c_str(),"wb");
if(!f)
return false;
int fd = fileno(f);
// set the headerData
HeaderData myHeaderData;
zlibDataFile = gzdopen(fd, "wb6");
// write the header data
//******* when I do that all previous data are erased !!!!
int err = gzwrite(zlibDataFile, (char*) &myHeaderData,
sizeof(HeaderData));
gzclose(zlibDataFile);
zlibDataFile = 0;
fclose(f);
}
Regards,
Paul D.
| |
|
| > // open the file for writing data
> FILE *f = fopen(path.c_str(),"wb");
Actually, that was my first mistake here "wb", but even with "ab", I
still have my data erased when I write my header info...
Paul D.
| |
| John Reiser 2005-07-27, 5:01 pm |
| > void writeData()
> {
> // open the file for writing data
> FILE *f = fopen(path.c_str(),"wb");
...
> fclose(f);
> }
>
> void writeHeaderData()
> {
> // open the file for writing header data at the top of the zlib file
> FILE *f = fopen(path.c_str(),"wb");
Already the file has been clobbered. "wb" means create a new file
for writing in binary mode, or truncate an existing file to zero length
for writing in binary mode.
Either: do not fclose() the file until after writing both the data
and the header [fs(f, 0, SEEK_SET) before writing the header];
or, use mode "r+b" to open the file [with no truncation of existing
contents] for both reading and writing the header.
--
| |
|
| Ok I rewrote my sample (code to below) to much simplier for testing.
Now gzdopen always return NULL though I have a valid FILE.
Does anyone get this pb ?
---------------------------------------
bool testZlib()
{
// open the file for writing
FILE *f = fopen("C:/temp/text.gz","wb");
if(!f)
return false;
int fd = fileno(f);
gzFile zlibFile = gzdopen(fd, "wb");
//****for some reason now gzdopen doesn't return a valid gzFile******/
if ( zlibFile == NULL )
return false; //exit here everytime
data d;
int err;
for (int i = 0; i<200; ++i)
{
err = gzwrite(zlibFile, (char*) &d, sizeof(data));
d.nb_records += 1;
}
gzclose(zlibFile);
zlibFile = 0;
// now we want to write again the first data.
fs (f,0,SEEK_SET);
fd = fileno(f);
zlibFile = gzdopen(fd, "wb");
if ( zlibFile == NULL )
return false;
err = gzwrite(zlibFile, (char*) &d, sizeof(data));
gzclose(zlibFile);
zlibFile = 0;
fclose(f);
return true;
}
-----------------
Thanks for your help
Paul D
| |
|
| Hi
I still didn't find a solution to my bug (issue with gzdopen) reported
in my previous post..
Is there something obvious I missed in my code ?
Thank you for your help
Paul D.
PD a =E9crit :
> Ok I rewrote my sample (code to below) to much simplier for testing.
> Now gzdopen always return NULL though I have a valid FILE.
>
> Does anyone get this pb ?
> ---------------------------------------
> bool testZlib()
> {
>
> // open the file for writing
> FILE *f =3D fopen("C:/temp/text.gz","wb");
> if(!f)
> return false;
>
> int fd =3D fileno(f);
> gzFile zlibFile =3D gzdopen(fd, "wb");
>
> //****for some reason now gzdopen doesn't return a valid gzFile******/
> if ( zlibFile =3D=3D NULL )
> return false; //exit here everytime
>
> data d;
> int err;
> for (int i =3D 0; i<200; ++i)
> {
> err =3D gzwrite(zlibFile, (char*) &d, sizeof(data));
> d.nb_records +=3D 1;
> }
>
>
> gzclose(zlibFile);
> zlibFile =3D 0;
>
>
> // now we want to write again the first data.
> fs (f,0,SEEK_SET);
> fd =3D fileno(f);
>
> zlibFile =3D gzdopen(fd, "wb");
> if ( zlibFile =3D=3D NULL )
> return false;
>
> err =3D gzwrite(zlibFile, (char*) &d, sizeof(data));
>
> gzclose(zlibFile);
> zlibFile =3D 0;
>
> fclose(f);
>
> return true;
> }
> -----------------
>=20
> Thanks for your help
>=20
> Paul D
| |
| Mark Adler 2005-07-31, 5:12 pm |
| PD wrote:
> I still didn't find a solution to my bug (issue with gzdopen) reported
> in my previous post..
> Is there something obvious I missed in my code ?
You are having issues with the basic use of stdio, not with zlib. I
recommend that you first get the code working with fdopen(), fwrite(),
and fclose(). Once that's working, then replace those with the gz*
counterparts.
mark
| |
| bios@ifrance.com 2005-08-02, 4:59 pm |
| Hello Mark,
So I did my homework....
---------------
int fd
fd = open("C:/temp/text.txt", O_WRONLY|O_CREAT);
FILE* file = fdopen(fd, "w");
char buffer[] = "This buffer contains 34 characters";
fwrite (buffer , 1 , 34 , File);
fclose (file );
--------------
It works good..no issue there !!!
Now...
----------------------------
int fd
fd = open("C:/temp/textb.txt.gz", O_WRONLY|O_CREAT);
gzFile file = gzdopen(fd, "w"); // ***** file = 0 after gzdopen with
a valid fd****//
char buffer[] = "This buffer contains 34 characters";
gzwrite (file , (char*) &buffer , 34 );
gzclose (file );
----------------------------
I can't get a gzFile using gzdopen.
Regards
Paul D.
| |
| Mark Adler 2005-08-03, 4:59 pm |
| bios@ifrance.com wrote:
> gzFile file = gzdopen(fd, "w"); // ***** file = 0 after gzdopen with
> a valid fd****//
That means that gzdopen() was not able to allocate enough memory for
the compression state. It needs about 256K bytes.
mark
| |
|
| Hello Mark,
Thanks for all the info so far...
Now what should I do to increase the memory space for the compression
state ?
Should I allocate some memory somewhere and tell gzdopen to use that
one ?
(gzopen works fine so it is able to allocate the memory)
Thanks
Paul D
| |
| Mark Adler 2005-08-19, 4:12 pm |
| PD wrote:
> Now what should I do to increase the memory space for the compression
> state ?
This is simply up to the malloc library of your compiler and whatever
limitations it imposes on when running on your machine.
mark
| |
| Robert Maas, see http://tinyurl.com/uh3t 2005-08-22, 3:55 am |
| > From: "PD" <b...@ifrance.com>
> I have some data that I can compress into a file with Zlib.
> Now, I'd like to have a header for this file that describes the data.
> Since the header information depends on the data, it means the header
> will need to be written in the file later on.
> How is it possible to write this block of data at the begin of the file
> with zlib ?
Is there any particular reason why that meta-data absolutely must be a
header (at start of file) instead of footer (just before end of file)?
Writing a footer is easier: First skip one word at start of file. Then
write compressed data. Then synchronized to next word boundary or
whatever correct boundary for footer would be, and remember where you
are right now. Then write meta-data as footer. Then close file.
Re-open file in overwrite mode, overwrite that one word at start of
file to be the location you remembered earlier, and immediately close
the file. Virtually no way to accidently trash the file, and no need
for zlib to be able to deal with random-access files.
If zlib doesn't allow random-access mode when decompressing the file
either, you do it like this: Open file in random-access mode, read that
one word at start, s to that address, read footer, close file.
Re-open file in regular zlib mode, skip one word, and use the
already-retrieved meta-data to decompress the file.
These are just ideas from a general think-outside-box attitude.
Whether what I suggest is possible in your situation I don't know.
Try them and see!
|
|
|
|
|