Home > Archive > Fortran > December 2005 > F95 I/O filesize problems
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
F95 I/O filesize problems
|
|
| stuart Lynn 2005-12-06, 7:05 pm |
| Hi all,
I am working with nag f95 and trying to read an unformatted file of
about 2-3Gb size. I basically have :
open(unit=1,file="/home/sl/galaxyCat/croton_big.dat",status="old",
form="unformatted")
do i =1,30
read (1,*) PosX
write(*,*) PosX
end do
which gives me the error of :
I/O error on unit 1: File too large
Program terminated by fatal I/O error
Abort
I have written the exact same program in f77 and it runs fine.
Has anyone come across this before or has any idea how I can fix it.
cheers
stuart
| |
| Arjen Markus 2005-12-06, 7:06 pm |
| It is the infamous 2GB limit imposed by the use of 4 byte integers in
older operating systems. Currently the operating system should be able
to handle larger files, but your compiler must be aware of it. Is there
a flag to indicate that you want big files?
(I do not know the NAG compiler - I only know of it).
Regards,
Arjen
| |
|
| Is the file formatted or not?
You open it as "unformatted" yet you read and write with format
specifications (the asterisk next to the right parentheses in the I/O
statements).
This won't get around the error message you currently have, but the
syntax of the I/O statements should match the OPEN.
| |
| Richard Maine 2005-12-06, 7:06 pm |
| jon <jon_d_r@msn.com> wrote:
> Is the file formatted or not?
>
> You open it as "unformatted" yet you read and write with format
> specifications (the asterisk next to the right parentheses in the I/O
> statements).
I almost overlooked this because I doubt it is your problem, but yes,
you have to get this right with any compiler. I am quite surprised by
the claim that it "runs fine" in "f77" (whatever "f77" means, which
isn't clear here). This is certainly not standard f77, though I suppose
you might have a compiler somewhere that allows it.
I regularly bug people about the misuse of the term "unformatted", and
this is exactly why. Misusing the term leads to writing code that won't
work. The standard uses the term "list-directed formatting" for what the
"*" signifies. One also sees it called things like "free" format. But it
isn't unformatted and won't work if you open the file as unformatted.
That being said, I doubt it is your only problem here (though it is at
least possible, and thus worth trying). Files larger than 2 gb (or 4gb
in some cases) are "problematic". Some operating systems and compilers
support them; others don't. Nag might possibly support such files on
some platforms, but not on all. This is the kind of thing that might
depend on exactly which Nag compiler you have (mostly which platform).
You'd have to ask NAG to know for sure.
--
Richard Maine | Good judgement comes from experience;
email: last name at domain . net | experience comes from bad judgement.
domain: summertriangle | -- Mark Twain
| |
|
| This may not exactly address your problem, but be aware that most
compilers will not address binary files larger than 2**31-1.
As an example:
integer*1 abyte
open(1,file=3Doutputfile,recl=3D1,form=3
D'binary',access=3D'direct')
write(3,rec=3Dn) abyte
The largest value of n can only be 2**31-1.
The only compiler that I have found that can exceed that limitation is
the Intel=AE Visual Fortran Compiler 9.0 for Windows (and its immediate
predecessor). I have not yet purchased the Intel compiler and I quote
only what I was told by Intel's tech support.
I currently use Lahey Fortran 95, version 5.6, so I have the same
problem.
http://www.intel.com/cd/software/pr...lers/219725.htm
Hope this helps.
Wayne
| |
|
| In my previous post the example should be:
open(1,.....
write(1,......
Sorry.
Wayne
| |
|
| > integer*1 abyte
> open(1,file=outputfile,recl=1,form='bina
ry',access='direct')
> write(3,rec=n) abyte
Please note that the correct way to write this is form='unformatted'.
> The largest value of n can only be 2**31-1.
>
> The only compiler that I have found that can exceed that limitation is
> the Intel® Visual Fortran Compiler 9.0 for Windows (and its immediate
> predecessor). I have not yet purchased the Intel compiler and I quote
> only what I was told by Intel's tech support.
I can confirm it's true for Intel 8.1 and 9.0 also.
malo /tmp $ cat a.f
integer*1 abyte
integer*8 n
n = huge(0_4)
n = 256 * n
print *, n
open(10,file="foo" ,recl=1,form='unformatted',access='direc
t')
write(10,rec=n) abyte
close(10)
end
malo /tmp $ ifort a.f && ./a.out
549755813632
malo /tmp $ ll foo
-rw-r--r-- 1 fx fx 2.0T Dec 7 00:27 foo
I guess 2.0 Tb should buy you some time...
> I currently use Lahey Fortran 95, version 5.6, so I have the same
> problem.
I can't say for Portland (stupid gateway computer is down again) but the
same is true of gfortran (I hope to fix this soon) and g95.
--
FX
| |
| Rich Townsend 2005-12-06, 7:06 pm |
| FX wrote:
>
>
> Please note that the correct way to write this is form='unformatted'.
Not if the OP meant 'threepwood'. By which I (and many others) mean some form of
bytestream I/O, where there are *NO* record length markers inserted by the
platform. I think there is something in F2003 that addresses Threepwood
functionality.
cheers,
Rich
| |
|
| FX:
You're correct about form='unformatted' in Lahey/Fujitsu Fortran 95
version 5.6.
Version 5.7 supports form='binary'. I jumped ahead of the product I
currently use.
http://www.lahey.com/lf9557.htm
Good night all.
Wayne
| |
| Joe Krahn 2005-12-06, 9:59 pm |
| FX wrote:
That only is a 2G file size limit if recl=1. So, can older compilers
handle larger files for larger values of recl?
[color=darkred]
>
>
> I can confirm it's true for Intel 8.1 and 9.0 also.
>
> malo /tmp $ cat a.f
> integer*1 abyte
> integer*8 n
> n = huge(0_4)
> n = 256 * n
> print *, n
> open(10,file="foo" ,recl=1,form='unformatted',access='direc
t')
> write(10,rec=n) abyte
> close(10)
> end
> malo /tmp $ ifort a.f && ./a.out
> 549755813632
> malo /tmp $ ll foo
> -rw-r--r-- 1 fx fx 2.0T Dec 7 00:27 foo
>
> I guess 2.0 Tb should buy you some time...
>
>
>
>
> I can't say for Portland (stupid gateway computer is down again) but the
> same is true of gfortran (I hope to fix this soon) and g95.
>
I just tried this with g95. It handles the large file OK. But, I noticed
that your file shows up as 2.0Tb, but should only be 512G.
Joe Krahn
| |
| Richard Maine 2005-12-07, 4:01 am |
| Joe Krahn <jkrahn@nc.rr.com> wrote:
> That only is a 2G file size limit if recl=1. So, can older compilers
> handle larger files for larger values of recl?
No. Regardless of record size, there are 32-bit internal variables in
the runtime libraries that have byte offsets in bytes (in many
implementations).
--
Richard Maine | Good judgement comes from experience;
email: last name at domain . net | experience comes from bad judgement.
domain: summertriangle | -- Mark Twain
| |
| stuart Lynn 2005-12-07, 7:59 am |
| Sorry yeah I didnt directly copy this from my code just wrote it out
again... put the *'s through force of habbit.
| |
| Paul Van Delst 2005-12-07, 7:04 pm |
| Joe Krahn wrote:
> I just tried this with g95. It handles the large file OK. But, I noticed
> that your file shows up as 2.0Tb, but should only be 512G.
I believe the default record length unit for intel is the word (4bytes) hence the 2.0Tb.
If the code is compiled with the "-assume byterecl" switch, then the resultant file should
be 512Gb (assuming your maths is correct... I didn't check).
cheers,
paulv
--
Paul van Delst
CIMSS @ NOAA/NCEP/EMC
| |
|
| The following e-mail is the latest word on this subject from
Lahey/Fujitsu Fortran 95 technical support:
Wayne,
You are partially right. In a patch (5.7e) we increased the file size
to 2**64-2 bytes (see description here
http://www.lahey.com/lf9557updt.htm). You will also note in that same
description it states that the maximum number of records is 2**32-1
(2,147,483,647) so for your purposes as you stated that you need direct
access of 1-byte binary files that still limits you to the 2**32-1 size
limit (2**32-1 * 1 byte = 2**32-1 bytes). If you can live with
sequential access or records larger than 1 byte then you can have
bigger files. Its just your specific use because of direct access with
a 1 byte record.
Cyrus
| |
| glen herrmannsfeldt 2005-12-08, 3:58 am |
| Richard Maine wrote:
> Joe Krahn <jkrahn@nc.rr.com> wrote:
[color=darkred]
> No. Regardless of record size, there are 32-bit internal variables in
> the runtime libraries that have byte offsets in bytes (in many
> implementations).
This is still true of many C libraries, which are sometimes used between
the Fortran library and the OS.
C allows ftell() and fs () to be used even on redirected stdout,
and some libraries protect against file write wraparound by not allowing
more than 2G-1 unless the executable has a special LARGEFILE attribute.
Others just don't allow it at all.
-- glen
|
|
|
|
|