File size limit exceeded error

Hi,

Here is the config of my system :
OS : Linux 2.4.19 (Redhat 7.3)
Cluster : Beowulf, 24 node, 48 processor
Compiler : pgf90

My code dumps unformatted data to a file.  As soon as my dump file size
reaches 2147483647 (i.e. 2^31-1).  I find that all my codes crash with a
"File size limit exceeded" error.

2^31 - 1 is probably largest integer in fortran and so my code crashes.
But, I still have some questions :

a) Why does fortran need to know the file size ? Cant it just dump the
data irrespective of what the file size is ?
b) Is there any way of circumventing this problem ?

Thanks,
shriram.

<dont@spam.me> wrote:
>Hi,
>
>Here is the config of my system :
>OS : Linux 2.4.19 (Redhat 7.3)
>Cluster : Beowulf, 24 node, 48 processor
>Compiler : pgf90
>
>My code dumps unformatted data to a file.  As soon as my dump file size
>reaches 2147483647 (i.e. 2^31-1).  I find that all my codes crash with a
>"File size limit exceeded" error.
>
>2^31 - 1 is probably largest integer in fortran and so my code crashes.

You can specify an integer with a range larger than the usual default 2^31
- 1 using the KIND function, possibly in conjunction with the SELECTED_INT_K
IND
function, as shown in the code below:

program xint_kind
integer         :: i
integer(kind=8) :: j
integer(kind=selected_int_kind(18)) :: k
print*,huge(i),huge(j),huge(k)
print*,kind(i),kind(j),kind(k)
end program xint_kind

which gives output

2147483647   9223372036854775807   9223372036854775807
4           8           8

using Compaq Visual Fortran 6.6 or Lahey/Fujitsu Fortran 95 5.70c

I'll let someone more qualified answer your file size question.



----== Posted via Newsfeed.Com - Unlimited-Uncensored-Secure Usenet News==--
--
http://www.newsfeed.com The #1 Newsgroup Service in the World! >100,000 News
groups
---= 19 East/West-Coast Specialized Servers - Total Privacy via Encryption =
---

dont@spam.me wrote:
> Here is the config of my system :
> OS : Linux 2.4.19 (Redhat 7.3)
> Cluster : Beowulf, 24 node, 48 processor
> Compiler : pgf90
>
> My code dumps unformatted data to a file.  As soon as my dump file size
> reaches 2147483647 (i.e. 2^31-1).  I find that all my codes crash with a
> "File size limit exceeded" error.

It's an OS thing (OK, it might be a compiler thing, too, but mostly it's
an OS thing);  it needs more than 32 bits to store file sizes and
offsets, which affects structures and typedefs.  I don't remember the
details, but as I recall, Redhat 7.x doesn't handle this situation
gracefully (if it handles it at all).  Plan on upgrading to at least
Redhat 8.

If you don't mind splitting your data into multiple files, you might
want to do that as an alternative to dealing with very large files.

Louis Krupp

dont@spam.me wrote:

> Here is the config of my system :
> OS : Linux 2.4.19 (Redhat 7.3)
> Cluster : Beowulf, 24 node, 48 processor
> Compiler : pgf90

> My code dumps unformatted data to a file.  As soon as my dump file size
> reaches 2147483647 (i.e. 2^31-1).  I find that all my codes crash with a
> "File size limit exceeded" error.


There are two things.

1)  Some file systems don't have a way to store files larger
than 2GB.

2)  Some libraries don't know how to write to them reliably.
C traditionally has used an int for the file pointer,
(argument of fs(), and return value of ftell().)
On some OS, when writing to a sable device, even if
the program doesn't actually use fs(), the OS/library
will refuse to write anymore.   Consider that a program
unaware that it was writing such a file could use fs()
with a position that had wrapped, writing over important
data.   Some have considered a fatal error better than
overwriting data.

Note that the Fortran library may use the C library to
actually perform I/O, so you may have C library limitations.

-- glen

For Portland group you need to invoke the "Large File Support Flag".
Compile your code with the -Mlfs flag.  ie.

pgf90 bigfile.f90 -o bigfile -Mlfs

GC




On Sun, 12 Sep 2004 11:08:53 -0500, <dont@spam.me> wrote:

>Hi,
>
>Here is the config of my system :
>OS : Linux 2.4.19 (Redhat 7.3)
>Cluster : Beowulf, 24 node, 48 processor
>Compiler : pgf90
>
>My code dumps unformatted data to a file.  As soon as my dump file size
>reaches 2147483647 (i.e. 2^31-1).  I find that all my codes crash with a
>"File size limit exceeded" error.
>
>2^31 - 1 is probably largest integer in fortran and so my code crashes.
>But, I still have some questions :
>
>a) Why does fortran need to know the file size ? Cant it just dump the
>   data irrespective of what the file size is ?
>b) Is there any way of circumventing this problem ?
>
>Thanks,
>shriram.
>

On Tue, 14 Sep 2004 03:39:31 GMT, glen herrmannsfeldt
<gah@ugcs.caltech.edu> wrote:

> dont@spam.me wrote:
> 
> 
>
>
> There are two things.
>
> 1)  Some file systems don't have a way to store files larger
> than 2GB.
>
Although I believe at least recent Linux filesystems can; I don't know
about the particular version cited.

> 2)  Some libraries don't know how to write to them reliably.
>      C traditionally has used an int for the file pointer,
>      (argument of fs(), and return value of ftell().)

s/int/long/, which is required to be at least 31 bits plus sign, and
traditionally is exactly that. C int can be as little as 15 bits plus
sign, and was on original (PDP-11) Unix, and that was not adequate for
file sizes even of that time (early 1970s).


- David.Thompson1 at worldnet.att.net

Dave Thompson wrote:

(snip on 2GB file limit)
 

> s/int/long/, which is required to be at least 31 bits plus sign, and
> traditionally is exactly that. C int can be as little as 15 bits plus
> sign, and was on original (PDP-11) Unix, and that was not adequate for
> file sizes even of that time (early 1970s).

Oops, yes.  I have used sizeof(int)==sizeof(long) machines
long enough that I forgot that.

The thing about this problem, at least on some OS, is that
it occurs even if the program doesn't use fs()/ftell(),
as long as it could use it.   Redirecting stdout to a file,
it applies.  Piping the output it doesn't.  I once did.

cat infile | some_program | dd of=outfile

to get around the problem.   I believe that was Solaris, though
I don't know if it is specific to Solaris.

-- glen