Home > Archive > Fortran > June 2005 > writing data to file
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
writing data to file
|
|
| icymist 2005-06-07, 4:02 pm |
| Hi all,
I have a basic doubt.
I use a code that generates a lot of date in every step of it's
execution.
I would like to know if it is faster to store all this data for say
some 100 steps and then write out the data into a file or will it be
faster to write out the data every step. Or won't it make any
difference.
Different from the above question, I can write the data in either
unformatted binary form, or formatted form and then compress the data
using gzip. What are the advantages or the di vantages of using one
over the other.
Thanks for any help.
Regards,
Chaitanya.
| |
| Richard E Maine 2005-06-07, 4:02 pm |
| In article <1118153603.991842.134580@g47g2000cwa.googlegroups.com>,
"icymist" <icymist@gmail.com> wrote:
> I would like to know if it is faster to store all this data for say
> some 100 steps and then write out the data into a file or will it be
> faster to write out the data every step. Or won't it make any
> difference.
It depends on details such as how much data per step, but as a broad
generalization, it is faster to write it out all at once because there
is a certain amount of overhead per write. It may well be, however, that
the difference is negligible, depending on details.
> Different from the above question, I can write the data in either
> unformatted binary form, or formatted form and then compress the data
> using gzip. What are the advantages or the divantages of using one
> over the other.
Well, if this is related to the first question at all... that is if you
care about the time it takes... formatted I/O is hugely slower than
unformatted (I prefer to avoid the term "binary", as base 2 doesn't
really have anything to do with the matter, and that's what "binary"
means). The difference in speed between formatted and unformatted is
likely to be far larger than any differences from, for example, writing
data every step vs writing in 100-step blocks. My rule of thumb is that
formatted I/O is typically about a factor of 10 slower.
Another divantage of formatted, which might or might not be important
to your application, is that it is non-trivial to write formatted data,
read it back, and get exactly the same value as the original, to the
last bit. If you care about the last bit, then unformatted is a far
better choice.
A divantage of unformatted is that it is machine-dependent. You have
to worry about such things as byte-sex and other issues to transfer
unformatted files between machines.
A divantage of using a separate gzip process is mostly that it is a
separate process. This can be operationally bothersome. Also, you
normally need room temporarily for both the compressed and uncompressed
files. One can integrate gzipping into writing (using some 3rd-party
libraries), but that introduces multiple extra complications.
--
Richard Maine | Good judgment comes from experience;
email: my first.last at org.domain | experience comes from bad judgment.
org: nasa, domain: gov | -- Mark Twain
|
|
|
|
|