Fortran - File IO

Author

File IO

ajs

2005-02-22, 4:01 pm

Hi
I am using INTEL 8.0 on a linux machine. I had a problem, managed to
fix it, but am trying to understand why this happens and if I should be
filing a bug report or something.

THe problem is with using implicit do loops to read : I am reading a
very large file (380MB or so) and if I read it using : (assuimg
im,jm,km,nm have been set)

read(10) fun(1:im,1:jm,1:km,1:nm)

it blows stack. So I set my stacksize limit to unlimited and re-rerun.
This works for one file. When I do the same read on a series of files
over and over again (in a loop, not multiple invocations of the
program), it eventually does the same thing! Then I changed the read
statement to :

read(10) ((((fun(i,j,k,n),i=1,im),j=1,jm),k=1,km)
,n=1,nm)

This magically works fine for virtually any number of reads. Is this
bug in the compiler or is this supposed to happen? Or is my stack
blowing theory itself all wrong? If that is the case, why did it work
on one file when I increased the stacksize limit?

Any help is appreciated, Thanks a lot.
Srinivasan

Richard E Maine

2005-02-22, 8:59 pm

In article <1109096956.623938.263090@l41g2000cwc.googlegroups.com>,
"ajs" <ajs00g@gmail.com> wrote:

> THe problem is with using implicit do loops to read

(Actually, from your description, the implicit DO loops are working
fine; your problem is when you read array slices instead of implicit DO
loops).

> read(10) fun(1:im,1:jm,1:km,1:nm)
>
> it blows stack. So I set my stacksize limit to unlimited and re-rerun.
> This works for one file. When I do the same read on a series of files
> over and over again (in a loop, not multiple invocations of the
> program), it eventually does the same thing! Then I changed the read
> statement to :
>
> read(10) ((((fun(i,j,k,n),i=1,im),j=1,jm),k=1,km)
,n=1,nm)
>
> This magically works fine for virtually any number of reads. Is this
> bug in the compiler or is this supposed to happen?...

First, be aware that "supposed to happen" is not the opposite of "bug in
the compiler". There are plenty of undesirable things that are neither
bugs in the compiler nor "supposed" to happen, but sometimes happen
anyway. Running into system limits is a classic example of that.

I can't commit to a diagnosis without seeing at least complete context
and preferably a complete runnable sample. But with that caveat in mind,
this has at least the sound of a compiler memory leak, which would be a
reportable bug. So I think you have probably found a bug, but I can't
stand behind that guess without more data.

As to why one form might leak, but not the other... well I could
speculate on possible implementation strategies, but it would be pure
speculation. Let's just leave it with the fact that implied DO loops can
be quite general and do things that are not possible with array slices;
this could quite likely mean that different implementation strategies
were used, which in turn would be subject to different possible bugs.

--
Richard Maine | Good judgment comes from experience;
email: my first.last at org.domain | experience comes from bad judgment.
org: nasa, domain: gov | -- Mark Twain

ajs

2005-02-22, 8:59 pm

THanks,
But is my diagnosis ot stack correct? I can include the code here for
people to see but then the problem doesnt reccur with smaller files
(even though I have experimented with a VERY large number of files , >
2000 files even) So I am not sure if anyone can check it, but here it
is anyways (the problem is in the read statement on line 238)

----------------
module datatypes
type,public :: block
integer :: imax,jmax,kmax
real,pointer,dimension(:,:,:) :: xc,yc,zc
real,pointer,dimension(:,:,:,:) :: fun
real,pointer,dimension(:,:,:) :: mean,rms
real,pointer,dimension(:,:,:) :: rstress
end type block
end module datatypes
program mkavg
use datatypes

implicit none

integer :: i,j,k,n,ierr,jerr,l,kerr,nf,m,mf
integer :: im,jm,km
integer :: nblks,numfiles,nfuns,nfuns_max,narg
character(len=100) :: filelist, filename
character(len=20),dimension(99) :: funname
character(len=1) :: ans
logical :: fexist, funnamed
real :: rfac
integer :: indxu,indxv,indxw,nu,nv,nw
logical :: do_rstress

integer,external :: iargc

type(block),allocatable,dimension(:) :: blk

real,pointer,dimension(:,:,:) :: mean,rms,rstress
real,pointer,dimension(:,:,:,:) :: fun
real,pointer,dimension(:,:,:) :: x,y,z
integer,pointer,dimension(:) :: funlist
!!======================================
===============
!! File containing Function File Names
!!======================================
===============

filelist = 'file.list'
narg = iargc()
if(narg .eq. 2) then
call getarg(1,filename)
if(filename .ne. '-i') stop "usage : mkavg.x -i files.list"
call getarg(2,filename)
filelist = trim(filename)
else
stop "usage : mkavg.x -i files.list"
end if

inquire(file = filelist, exist = fexist)
if(.not. fexist) stop 'No File List Found'

!!======================================
===============
!!Read in the XYZ grid
!!======================================
===============

inquire(file = 'recomp.craft.xyz', exist=fexist)
if(.not. fexist) stop 'Need XYZ file, Cannot Proceed'

open(10, file = 'recomp.craft.xyz', form = 'unformatted')
read(10,iostat=jerr) i,j,k
if(jerr == 0) then
nblks = 1
rewind(10)
else
rewind(10)
read(10,iostat=kerr) nblks
if(kerr .ne. 0) then
print*, "Error Reading File : ", trim(filename), " Skipping"
close(10)
endif
endif

allocate(blk(nblks))
read(10)(blk(n)%imax,blk(n)%jmax,blk(n)%
kmax,n=1,nblks)

do n = 1, nblks
i = blk(n)%imax; j = blk(n)%jmax; k = blk(n)%kmax
allocate(blk(n)%xc(1:i,1:j,1:k))
allocate(blk(n)%yc(1:i,1:j,1:k))
allocate(blk(n)%zc(1:i,1:j,1:k))
read(10) blk(n)%xc(1:i,1:j,1:k), &
blk(n)%yc(1:i,1:j,1:k), &
blk(n)%zc(1:i,1:j,1:k)
enddo
close(10)

!!======================================
===============
!! Write out Averaged Coordinate File
!!======================================
===============

open(10, file = 'avg.xyz', form = 'unformatted')
write(10)nblks
write(10)(blk(n)%imax,blk(n)%jmax,1,n=1,
nblks)
do n = 1,nblks
i = blk(n)%imax; j = blk(n)%jmax
write(10)blk(n)%xc(1:i,1:j,1), &
blk(n)%yc(1:i,1:j,1), &
blk(n)%zc(1:i,1:j,1)
enddo
close(10)
print*, "Done Creating Single Plane Grid File"

do n = 1, nblks
deallocate(blk(n)%xc,blk(n)%yc,blk(n)%zc
)
end do

!!======================================
===============
!! Read in craft.nam to get fun names
!!======================================
===============
nfuns = 1
inquire(file = 'craft.nam', exist=fexist)
if(fexist) then
open(10, file = 'craft.nam')
ierr = 0
n = 1
do while(ierr == 0)
read(10,'(a)',iostat=ierr) funname(n)
if(ierr .ne. 0) exit
n = n + 1
end do
close(10)

nfuns_max = n-1
print*, "Found ", nfuns_max, " Variables "
do n = 1, nfuns_max
write(*,'(i2,a,a)')n,". ", trim(funname(n))
end do
print*, "Select Ones to Use for Averaging : "
else
print*, "Could Not find Function List, Doing Blind Numbers "
do n = 1, 99
write(funname(n),'(a,i3.3)')"fun_",n
end do
end if
write(*,1) "Enter Number of Functions to be Averaged : "
read*, nfuns
allocate(funlist(nfuns))
do n = 1, nfuns
write(*,'(a,i3,a,$)')" Enter Var. Number ", n, " : "
read*, funlist(n)
end do

if(fexist) then
indxu = -1; indxv = -1; indxw = -1
do_rstress = .false.
do n = 1, nfuns
if(index(funname(funlist(n)),'u (m/sec)') .gt. 0) then
nu = n
indxu = funlist(n)
end if
if(index(funname(funlist(n)),'v (m/sec)') .gt. 0) then
nv = n
indxv = funlist(n)
end if
if(index(funname(funlist(n)),'w (m/sec)') .gt. 0) then
nw = n
indxw = funlist(n)
end if
end do
if(indxu > 0 .and. indxv > 0 .and. indxw > 0) do_rstress = .true.
if(.not. do_rstress) then
print*, " Did not find all velocities in Function list :"
print*, " -- no Reynolds Stress for you! ;) "
end if
else
write(*,1)"Do you want Reynolds Stresses : "
read(*,'(a)') ans
if(ans .eq. 'y' .or. ans == 'Y') then
write(*,1)"Enter Function Number for U : "
read*,indxu
write(*,1)"Enter Function Number for V : "
read*,indxv
write(*,1)"Enter Function Number for w : "
read*,indxw
else
indxu = -1; indxv = -1; indxw = -1
do_rstress = .false.
end if
end if

!!======================================
===============
!! File Loop - Read in each file and Compute mean and rms
!!======================================
===============

open(10, file = filelist, position = 'rewind')
numfiles = 0
fileloop1 : do
read(10,fmt='(a)',iostat=ierr) filename
if(ierr .lt. 0) then
print*, "Completed reading : ", trim(filelist)
print*, " Total Number of Files Read = ", numfiles
exit fileloop1
else if(ierr .gt. 0) then
print*, "Error reading : ", trim(filelist), numfiles
exit fileloop1
endif
numfiles = numfiles + 1

print*, "Reading Function File ", trim(filename)

!!======================================
===============
!! Read Function File
!! -- Number of Blocks
!!======================================
===============

open(11, file = trim(filename), form = 'unformatted')
if(nblks .eq. 1) then
read(11,iostat=jerr)(blk(n)%imax,blk(n)%
jmax,blk(n)%kmax,nf, n = 1,
nblks)
else
read(11,iostat=jerr) nblks
read(11,iostat=jerr)(blk(n)%imax,blk(n)%
jmax,blk(n)%kmax,nf, n = 1,
nblks)
end if
if(jerr .ne. 0) then
print*, "......Error Reading imax,jmax,kmax : ", trim(filename), "
Skipping"
numfiles = numfiles - 1
close(11)
cycle fileloop1
endif
if(nfuns == -1) nfuns = nf
print*, "----Number of Functions in file = ", nf

!!======================================
===============
!! -- Block Functions
!!======================================
===============

if(numfiles == 1) then
do n = 1, nblks
i = blk(n)%imax; j = blk(n)%jmax; k = blk(n)%kmax
allocate(blk(n)%fun(nf,1:i,1:j,1:k))
enddo
endif

do n = 1, nblks
im= blk(n)%imax; jm= blk(n)%jmax; km= blk(n)%kmax
read(11,iostat=jerr)
((((blk(n)%fun(l,i,j,k),i=1,im),j=1,jm),
k=1,km),l=1,nf)
if(jerr .ne. 0) then
print*, "......Error Reading Functions : ", trim(filename), "
Skipping"
numfiles = numfiles - 1
close(11)
cycle fileloop1
endif
enddo
close(11)
print*, "----Read in File Successfully"

!!======================================
===============
!! Mean and RMS for each Function
!! Note that mean is time and z averaged
!!======================================
===============

if(numfiles == 1) then
do n = 1, nblks
i = blk(n)%imax; j = blk(n)%jmax
allocate(blk(n)%mean(nfuns,1:i,1:j))
allocate(blk(n)%rms(nfuns,1:i,1:j))
if(do_rstress) allocate(blk(n)%rstress(3,1:i,1:j))
blk(n)%mean = 0.0
blk(n)%rms = 0.0
if(do_rstress) blk(n)%rstress = 0.0
enddo
endif

do n = 1, nblks
mean => blk(n)%mean
fun => blk(n)%fun
rms => blk(n)%rms
if(do_rstress) rstress => blk(n)%rstress

do k = 1, blk(n)%kmax
i = blk(n)%imax; j = blk(n)%jmax

do m = 1, nfuns
mf = funlist(m)
mean(m,1:i,1:j) = mean(m,1:i,1:j) + fun(mf,1:i,1:j,k)
rms(m,1:i,1:j) = rms(m,1:i,1:j) + fun(mf,1:i,1:j,k)**2
end do
if(do_rstress) rstress(1,1:i,1:j) = rstress(1,1:i,1:j) +
fun(indxu,1:i,1:j,k)*fun(indxv,1:i,1:j,k
)
if(do_rstress) rstress(2,1:i,1:j) = rstress(2,1:i,1:j) +
fun(indxu,1:i,1:j,k)*fun(indxw,1:i,1:j,k
)
if(do_rstress) rstress(3,1:i,1:j) = rstress(3,1:i,1:j) +
fun(indxv,1:i,1:j,k)*fun(indxw,1:i,1:j,k
)

enddo
enddo

enddo fileloop1
close(10)

if(numfiles == 0) stop ' No Function File List Found '

print*, "Completed reading function files : ", numfiles

do n = 1, nblks
i = blk(n)%imax; j = blk(n)%jmax; k = blk(n)%kmax
mean => blk(n)%mean
fun => blk(n)%fun
rms => blk(n)%rms
if(do_rstress) rstress => blk(n)%rstress
rfac = 1.0 / float(k*numfiles)

mean(1:nfuns,1:i,1:j) = mean(1:nfuns,1:i,1:j) * rfac
rms(1:nfuns,1:i,1:j) = rms(1:nfuns,1:i,1:j) * rfac
if(do_rstress) rstress(1:3,1:i,1:j) = rstress(1:3,1:i,1:j) * rfac

rms(1:nfuns,1:i,1:j) = rms(1:nfuns,1:i,1:j) -
mean(1:nfuns,1:i,1:j)**2
rms(1:nfuns,1:i,1:j) = sqrt(abs(rms(1:nfuns,1:i,1:j)))

if(do_rstress) rstress(1,i:i,1:j) = rstress(1,1:i,1:j) -
mean(nu,1:i,1:j)*mean(nv,1:i,1:j)
if(do_rstress) rstress(2,i:i,1:j) = rstress(2,1:i,1:j) -
mean(nu,1:i,1:j)*mean(nw,1:i,1:j)
if(do_rstress) rstress(3,i:i,1:j) = rstress(3,1:i,1:j) -
mean(nv,1:i,1:j)*mean(nw,1:i,1:j)
enddo

nf = 2*nfuns
if(do_rstress) nf = nf + 3
open(10, file = 'avg.fun', form = 'unformatted')
write(10)nblks
write(10)(blk(n)%imax,blk(n)%jmax,1,nf,n
=1,nblks)
do n = 1,nblks
i = blk(n)%imax; j = blk(n)%jmax; k = blk(n)%kmax
if(do_rstress) then
write(10)(blk(n)%mean(l,1:i,1:j),l=1,nfu
ns), &
(blk(n)%rms(l,1:i,1:j),l=1,nfuns), &
(blk(n)%rstress(l,1:i,1:j),l=1,nfuns)
else
write(10)(blk(n)%mean(l,1:i,1:j),l=1,nfu
ns), &
(blk(n)%rms(l,1:i,1:j),l=1,nfuns)
end if
enddo
close(10)

do n = 1, nfuns
mf = funlist(n)
l = len_trim(funname(mf))
j = index(funname(mf)(1:l)," ")
if(j .gt. 0) funname(mf) = funname(mf)(1:j-1)
end do

open(10, file = 'avg.res')
write(10,*) nf,0,0
write(10,*) 1
write(10,*) 0.0
do n = 1, nfuns
mf = funlist(n)
write(10,'(a,i3,a,a,a)')'avg.fun F ',n, " ",trim(funname(mf)),"_bar"
end do
do n = 1, nfuns
mf = funlist(n)
write(10,'(a,i3,a,a,a)')'avg.fun F ',n+nfuns, "
",trim(funname(mf)),"_rms"
end do
if(do_rstress) then
write(10,'(a,i3,a)')'avg.fun F ',2*nfuns+1, " up_vp"
write(10,'(a,i3,a)')'avg.fun F ',2*nfuns+2, " up_wp"
write(10,'(a,i3,a)')'avg.fun F ',2*nfuns+3, " vp_wp"
end if

close(10)

1 format(a,$)

stop

contains

real function viscosity(temp) result(zmu)
implicit none

real :: temp

zmu = 1.458e-6 * temp**1.5/(temp+110.4)

end function viscosity
end program mkavg
--------------------------------

Dick Hendrickson

2005-02-22, 8:59 pm

Sorry, I lost the original post, so I'll put my replies to
AJS in Richard's comments.

Richard E Maine wrote:
> In article <1109096956.623938.263090@l41g2000cwc.googlegroups.com>,
> "ajs" <ajs00g@gmail.com> wrote:
>
>
>
>
> (Actually, from your description, the implicit DO loops are working
> fine; your problem is when you read array slices instead of implicit DO
> loops).
>
>

A reasonable guess is that the compiler creates a
temporary space on the stack for the fun array. It's
not obvious that im, jm, etc., are going to cover the
whole array, so the compiler might creat a temporary
and then copy that to fun after the read finishes.

If it doesn't release the stack (or even heap) space
after the read, that's probably a bug in the I/O design.
But, in your longer example (which was too long for me
to completely read ;) ), you had some reads with several
arrays (actually structure components that were arrays).
Even if the compiler releases the storage after the read,
needing several temporaries on one read could overflow
memory.
[color=darkred]

But, this is 40 year old fortran, and the compilers have all
learned how to do this very well. It's unlikely that they
will generate temporaries for fun. I'd guess that they
would do a bulk read of a 1 to im column at a time directly
into fun. No temporaries.

You could try writing a smaller sample program that just
write a ton of long records to a file and then tries to
read them in a loop. By varing record size and stack
size you might be able to figure out something. Or maybe
others could try it with different compilers and see if they
can figure out what is going on.

[color=darkred]
>
>
> First, be aware that "supposed to happen" is not the opposite of "bug in
> the compiler". There are plenty of undesirable things that are neither
> bugs in the compiler nor "supposed" to happen, but sometimes happen
> anyway. Running into system limits is a classic example of that.
>
> I can't commit to a diagnosis without seeing at least complete context
> and preferably a complete runnable sample. But with that caveat in mind,
> this has at least the sound of a compiler memory leak, which would be a
> reportable bug. So I think you have probably found a bug, but I can't
> stand behind that guess without more data.
>
> As to why one form might leak, but not the other... well I could
> speculate on possible implementation strategies, but it would be pure
> speculation. Let's just leave it with the fact that implied DO loops can
> be quite general and do things that are not possible with array slices;
> this could quite likely mean that different implementation strategies
> were used, which in turn would be subject to different possible bugs.
>

beliavsky@aol.com

2005-02-22, 8:59 pm

ajs wrote:
> THanks,
> But is my diagnosis ot stack correct? I can include the code here for
> people to see but then the problem doesnt reccur with smaller files
> (even though I have experimented with a VERY large number of files ,
>
> 2000 files even) So I am not sure if anyone can check it, but here it
> is anyways (the problem is in the read statement on line 238)
>
> ----------------
> module datatypes
> type,public :: block
> integer :: imax,jmax,kmax
> real,pointer,dimension(:,:,:) :: xc,yc,zc
> real,pointer,dimension(:,:,:,:) :: fun
> real,pointer,dimension(:,:,:) :: mean,rms
> real,pointer,dimension(:,:,:) :: rstress
> end type block
> end module datatypes

Intel Fortran 8.0 for Windows (and I assume for Linux, too) supports
the Fortran 2003 (also a Technical Report) feature of allocatable
components of derived types. Maybe using allocatable components instead
of pointer components would solve the problem, which is not to say that
the code with pointer components should not work.

Rich Townsend

2005-02-22, 8:59 pm

beliavsky@aol.com wrote:
> ajs wrote:
>
>
>
> Intel Fortran 8.0 for Windows (and I assume for Linux, too) supports
> the Fortran 2003 (also a Technical Report) feature of allocatable
> components of derived types. Maybe using allocatable components instead
> of pointer components would solve the problem, which is not to say that
> the code with pointer components should not work.
>

I wouldn't hold my breath; most of the bug reports I've submitted to
Intel, have been in relation to the allocatable components feature.

Of course, that's not to say that ifort isn't, IMHO, a great product; in
truth, most compiler vendors have had difficulty getting allocatable
components working correctly.

The OP may have more success in upgrading to the most recent version of
ifort; 8.1 has been out for some time now.

cheers,

Rich

Richard E Maine

2005-02-22, 8:59 pm

In article <1109102491.775131.64270@z14g2000cwz.googlegroups.com>,
"ajs" <ajs00g@gmail.com> wrote:

> But is my diagnosis ot stack correct? I can include the code here for
> people to see but then the problem doesnt reccur with smaller files
> (even though I have experimented with a VERY large number of files , >
> 2000 files even) So I am not sure if anyone can check it, but here it
> is anyways (the problem is in the read statement on line 238)
[code elided]

I suppose I asked for it (literally), so perhaps I should spend a little
time studying it. :-) ...

Looks to me like your diagnosis is correct. I didn't actually try to run
the code (much less with lots of big files) to duplicate the error, but
looking at the code, and adding your description of the symptoms, I
agree with your analysis.

I threw the code at 2 compilers that were handy at the moment (Nag and
g95), just to see what they would say about it - not that the reported
problems would likely show up at compile time anyway. Nothing relevant
shows up. Just FYI, although not related to this problem, NAG did
notice the nonstandard use of $ in formats, plus a few unused variables,
but nothing that I'd regard as serious problems.

--
Richard Maine | Good judgment comes from experience;
email: my first.last at org.domain | experience comes from bad judgment.
org: nasa, domain: gov | -- Mark Twain

Steve Lionel

2005-02-23, 3:59 am

ajs wrote:

> This magically works fine for virtually any number of reads. Is this
> bug in the compiler or is this supposed to happen? Or is my stack
> blowing theory itself all wrong? If that is the case, why did it work
> on one file when I increased the stacksize limit?

The version of the compiler you are using does not cut the stack back
after an I/O statement that creates a stack temporary. Current versions
(8.1.25 is recent) do.

Steve Lionel
Intel Corporation

Sponsored Links