Fortran - File size limit exceeded error

Author

File size limit exceeded error

2004-09-12, 3:57 pm

Hi,

Here is the config of my system :
OS : Linux 2.4.19 (Redhat 7.3)
Cluster : Beowulf, 24 node, 48 processor
Compiler : pgf90

My code dumps unformatted data to a file. As soon as my dump file size
reaches 2147483647 (i.e. 2^31-1). I find that all my codes crash with a
"File size limit exceeded" error.

2^31 - 1 is probably largest integer in fortran and so my code crashes.
But, I still have some questions :

a) Why does fortran need to know the file size ? Cant it just dump the
data irrespective of what the file size is ?
b) Is there any way of circumventing this problem ?

Thanks,
shriram.

beliavsky@aol.com

2004-09-12, 8:56 pm

<dont@spam.me> wrote:
>Hi,
>
>Here is the config of my system :
>OS : Linux 2.4.19 (Redhat 7.3)
>Cluster : Beowulf, 24 node, 48 processor
>Compiler : pgf90
>
>My code dumps unformatted data to a file. As soon as my dump file size
>reaches 2147483647 (i.e. 2^31-1). I find that all my codes crash with a
>"File size limit exceeded" error.
>
>2^31 - 1 is probably largest integer in fortran and so my code crashes.

You can specify an integer with a range larger than the usual default 2^31
- 1 using the KIND function, possibly in conjunction with the SELECTED_INT_KIND
function, as shown in the code below:

program xint_kind
integer :: i
integer(kind=8) :: j
integer(kind=selected_int_kind(18)) :: k
print*,huge(i),huge(j),huge(k)
print*,kind(i),kind(j),kind(k)
end program xint_kind

which gives output

2147483647 9223372036854775807 9223372036854775807
4 8 8

using Compaq Visual Fortran 6.6 or Lahey/Fujitsu Fortran 95 5.70c

I'll let someone more qualified answer your file size question.

----== Posted via Newsfeed.Com - Unlimited-Uncensored-Secure Usenet News==----
http://www.newsfeed.com The #1 Newsgroup Service in the World! >100,000 Newsgroups
---= 19 East/West-Coast Specialized Servers - Total Privacy via Encryption =---

Louis Krupp

2004-09-13, 3:57 am

dont@spam.me wrote:
> Here is the config of my system :
> OS : Linux 2.4.19 (Redhat 7.3)
> Cluster : Beowulf, 24 node, 48 processor
> Compiler : pgf90
>
> My code dumps unformatted data to a file. As soon as my dump file size
> reaches 2147483647 (i.e. 2^31-1). I find that all my codes crash with a
> "File size limit exceeded" error.

It's an OS thing (OK, it might be a compiler thing, too, but mostly it's
an OS thing); it needs more than 32 bits to store file sizes and
offsets, which affects structures and typedefs. I don't remember the
details, but as I recall, Redhat 7.x doesn't handle this situation
gracefully (if it handles it at all). Plan on upgrading to at least
Redhat 8.

If you don't mind splitting your data into multiple files, you might
want to do that as an alternative to dealing with very large files.

Louis Krupp

glen herrmannsfeldt

2004-09-14, 3:57 am

dont@spam.me wrote:

> Here is the config of my system :
> OS : Linux 2.4.19 (Redhat 7.3)
> Cluster : Beowulf, 24 node, 48 processor
> Compiler : pgf90

> My code dumps unformatted data to a file. As soon as my dump file size
> reaches 2147483647 (i.e. 2^31-1). I find that all my codes crash with a
> "File size limit exceeded" error.

There are two things.

1) Some file systems don't have a way to store files larger
than 2GB.

2) Some libraries don't know how to write to them reliably.
C traditionally has used an int for the file pointer,
(argument of fs

(), and return value of ftell().)
On some OS, when writing to a s

able device, even if
the program doesn't actually use fs

(), the OS/library
will refuse to write anymore. Consider that a program
unaware that it was writing such a file could use fs

()
with a position that had wrapped, writing over important
data. Some have considered a fatal error better than
overwriting data.

Note that the Fortran library may use the C library to
actually perform I/O, so you may have C library limitations.

-- glen

George Costanza

2004-09-14, 3:59 pm

For Portland group you need to invoke the "Large File Support Flag".
Compile your code with the -Mlfs flag. ie.

pgf90 bigfile.f90 -o bigfile -Mlfs

GC

On Sun, 12 Sep 2004 11:08:53 -0500, <dont@spam.me> wrote:

>Hi,
>
>Here is the config of my system :
>OS : Linux 2.4.19 (Redhat 7.3)
>Cluster : Beowulf, 24 node, 48 processor
>Compiler : pgf90
>
>My code dumps unformatted data to a file. As soon as my dump file size
>reaches 2147483647 (i.e. 2^31-1). I find that all my codes crash with a
>"File size limit exceeded" error.
>
>2^31 - 1 is probably largest integer in fortran and so my code crashes.
>But, I still have some questions :
>
>a) Why does fortran need to know the file size ? Cant it just dump the
> data irrespective of what the file size is ?
>b) Is there any way of circumventing this problem ?
>
>Thanks,
>shriram.
>

Dave Thompson

2004-09-20, 4:01 am

On Tue, 14 Sep 2004 03:39:31 GMT, glen herrmannsfeldt
<gah@ugcs.caltech.edu> wrote:

> dont@spam.me wrote:
>
>
>
>
> There are two things.
>
> 1) Some file systems don't have a way to store files larger
> than 2GB.
>
Although I believe at least recent Linux filesystems can; I don't know
about the particular version cited.

> 2) Some libraries don't know how to write to them reliably.
> C traditionally has used an int for the file pointer,
> (argument of fs

(), and return value of ftell().)

s/int/long/, which is required to be at least 31 bits plus sign, and
traditionally is exactly that. C int can be as little as 15 bits plus
sign, and was on original (PDP-11) Unix, and that was not adequate for
file sizes even of that time (early 1970s).

- David.Thompson1 at worldnet.att.net

glen herrmannsfeldt

2004-09-20, 4:01 am

Dave Thompson wrote:

(snip on 2GB file limit)

[color=darkred]
> s/int/long/, which is required to be at least 31 bits plus sign, and
> traditionally is exactly that. C int can be as little as 15 bits plus
> sign, and was on original (PDP-11) Unix, and that was not adequate for
> file sizes even of that time (early 1970s).

Oops, yes. I have used sizeof(int)==sizeof(long) machines
long enough that I forgot that.

The thing about this problem, at least on some OS, is that
it occurs even if the program doesn't use fs

()/ftell(),
as long as it could use it. Redirecting stdout to a file,
it applies. Piping the output it doesn't. I once did.

cat infile | some_program | dd of=outfile

to get around the problem. I believe that was Solaris, though
I don't know if it is specific to Solaris.

-- glen

Sponsored Links