Home > Archive > Fortran > February 2005 > Data saving/loading and binary file format
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Data saving/loading and binary file format
|
|
| s8ngsu3@yahoo.com 2005-02-22, 8:58 am |
| Hello,
binary files have been already discussed here many times, but I still
cannot devise an elegant solution to my problem. I need to save some
data from my program into a binary file (binary - to avoid round-off
and truncation errors when converting to/from text) and then to load
these data. E.g., if I have some variables, writing would look like
this:
integer i
real x
....
write (1) i
write (1) x
....
and reading:
read(1) i
read(1) x
....
The problem with this approach is that the sequence of read statements
must exactly match the sequence of write statements. If I need to
update the program to write some more data into the file, like
write(1) i
write(1) j
write(1) x
....
then I will have to rearrange the read statements accordingly and, even
worse, I won't be able to read old data files anymore. I planned to put
some header at
the beginning of the file that would contain info about which variable
is located where in the file (i.e., at what offset from the beginning
of the file), but, to implement this, I need direct-access files, and
AFAIK they use fixed record lengths, which is unsuitable if data of
various types must be stored. Is there some way to work with binary
file headers in Fortran95? Most standard file formats, like JPEG, PDF
etc. use headers. How are they handled in Fortran?
| |
| Pierre Asselin 2005-02-22, 4:01 pm |
| s8ngsu3@yahoo.com wrote:
> ...
> write (1) i
> write (1) x
> ...
> and reading:
> read(1) i
> read(1) x
> ...
> The problem with this approach is that the sequence of read statements
> must exactly match the sequence of write statements. If I need to
> update the program to write some more data into the file, like
> write(1) i
> write(1) j
> write(1) x
> ...
> then I will have to rearrange the read statements accordingly and, even
> worse, I won't be able to read old data files anymore. I planned to put
> some header at
> the beginning of the file that would contain info about which variable
> is located where in the file (i.e., at what offset from the beginning
> of the file), but, to implement this, I need direct-access files, [ ... ]
Turn your thinking around. Instead of a map from variable to file
offset, use a map from file offset to variable. For example, you
could write before each variable a record identifying the variable.
integer, parameter:: C_I= 1, C_X= 2
write(1) C_I; write(1) i
write(1) C_X; write(1) x
and to read,
integer code
do
read(1, err=99) code
select case(code)
case(C_I); read(1) i
case(C_X); read(1) x
end select
end do
99 continue
(not tested)
Later you add a parameter C_J and a case clause to handle reading
a j. The C_* can never change once you introduce them, they're a
part of your file interface. You can only add new ones until you
run out --but that will take a while.
Hopefully the payload for each C_* will be more than a single
variable, for size and efficiency reason.
--
pa at panix dot com
| |
| s8ngsu3@yahoo.com 2005-02-22, 4:01 pm |
| Pierre Asselin wrote:
>
> Turn your thinking around. Instead of a map from variable to file
> offset, use a map from file offset to variable. For example, you
> could write before each variable a record identifying the variable.
Thank you, that's an interesting idea (if one doesn't mind bloating the
files). But anyway, how do they handle these standard binary files that
contain headers?
| |
| Richard E Maine 2005-02-22, 4:01 pm |
| In article <1109083224.257551.60310@l41g2000cwc.googlegroups.com>,
s8ngsu3@yahoo.com wrote:
> Thank you, that's an interesting idea (if one doesn't mind bloating the
> files). But anyway, how do they handle these standard binary files that
> contain headers?
You implied that you've read the many discussions here of "binary" (a
misnomer for reasons I've mentioned before and so I don't repeat here)
files. If so, perhaps you missed one of the main points made in many of
those threads - that Fortran 2003 adds a feature for dealing with them,
but that it is darned awkward in previous versions of the standard.
There are plenty of compiler-specific options for dealing with them in
f90/f95. Almost every compiler has something, but the simple options
aren't portable, and the portable options aren't simple. The f2003
feature was added to address a need here.
That being said, the big problems are when you have to deal with file
formats defined elsewhere. When you are defining the file format, you
can usually design one that avoids the worst of the problems. But for
things like what you call "standard binary files", it often turns out to
be worth using the compiler-specific options in f90/f95 for now.
--
Richard Maine | Good judgment comes from experience;
email: my first.last at org.domain | experience comes from bad judgment.
org: nasa, domain: gov | -- Mark Twain
| |
| Paul Van Delst 2005-02-22, 4:01 pm |
| s8ngsu3@yahoo.com wrote:
> Hello,
>
> binary files have been already discussed here many times, but I still
> cannot devise an elegant solution to my problem. I need to save some
> data from my program into a binary file (binary - to avoid round-off
> and truncation errors when converting to/from text) and then to load
> these data. E.g., if I have some variables, writing would look like
> this:
>
> integer i
> real x
> ...
> write (1) i
> write (1) x
> ...
>
> and reading:
>
> read(1) i
> read(1) x
> ...
>
> The problem with this approach is that the sequence of read statements
> must exactly match the sequence of write statements. If I need to
> update the program to write some more data into the file, like
>
> write(1) i
> write(1) j
> write(1) x
> ...
>
> then I will have to rearrange the read statements accordingly and, even
> worse, I won't be able to read old data files anymore. I planned to put
> some header at
> the beginning of the file that would contain info about which variable
> is located where in the file (i.e., at what offset from the beginning
> of the file), but, to implement this, I need direct-access files, and
> AFAIK they use fixed record lengths, which is unsuitable if data of
> various types must be stored. Is there some way to work with binary
> file headers in Fortran95? Most standard file formats, like JPEG, PDF
> etc. use headers. How are they handled in Fortran?
I would suggest you look into the netCDF file format, which has a Fortran90 API (along
with the f77 and C one).
Check out
http://my.unidata.ucar.edu/content/software/netcdf
Installation and use is relatively simple (and there is plenty of build info on that site
too. e.g. environment variables etc)
cheers,
paulv
--
Paul van Delst
CIMSS @ NOAA/NCEP/EMC
| |
| s8ngsu3@yahoo.com 2005-02-22, 4:01 pm |
| Richard E Maine wrote:
>
> You implied that you've read the many discussions here of "binary" (a
> misnomer for reasons I've mentioned before and so I don't repeat
here)
> files. If so, perhaps you missed one of the main points made in many
of
> those threads - that Fortran 2003 adds a feature for dealing with
them,
> but that it is darned awkward in previous versions of the standard.
I don't have any Fortran2003 compiler yet.
> There are plenty of compiler-specific options for dealing with them
in
> f90/f95. Almost every compiler has something, but the simple options
> aren't portable, and the portable options aren't simple.
I would prefer to avoid any compiler-specific options, because I want
my program to be compilable by different compilers and run on different
OSes.
| |
| s8ngsu3@yahoo.com 2005-02-22, 4:01 pm |
| Paul Van Delst wrote:
> I would suggest you look into the netCDF file format, which has a
Fortran90 API
Thank you, I'll try it out.
| |
| Richard E Maine 2005-02-22, 4:01 pm |
| In article <1109092478.251042.225520@f14g2000cwb.googlegroups.com>,
s8ngsu3@yahoo.com wrote:
> I don't have any Fortran2003 compiler yet.
Nor does anyone else. :-)
> Richard E Maine wrote:
>
> I would prefer to avoid any compiler-specific options, because I want
> my program to be compilable by different compilers and run on different
> OSes.
Absolutely. I 100% agree. That's why I was such an advocate of the f2003
feature to solve this issue in the future.
But for today, that's just the way life is. If you want a solution that
is simple, portable, and available today, it doesn't exist. My
preferences agree with yours here, but that doesn't change the facts
today (though it helps in changing them for the future).
Fortunately for your immediate problem, you get to design the file, so
you do have enough options to meet the stated needs today (see posts by
others for at least one suggestion). If you needed to deal with
outside-defined formats, then you'd have to give up at least one of your
(and my) preferences - simple, portable, or now.
--
Richard Maine | Good judgment comes from experience;
email: my first.last at org.domain | experience comes from bad judgment.
org: nasa, domain: gov | -- Mark Twain
|
|
|
|
|