Need help with large file reading and writing.

Hello Pals:

When I use fortran 90 code to read larger files,about 1G each, and
write the data to other opened files with 32-bit Intel-compiler. Each
time, I just opened 3 files simultaneously and closed unused ones. The
code always stops at the point of 10 files reading. I am not sure it is
the compiler problem about the limitation of buffer memory.

Thanks very much

Liscon


Simple code:

open(unit=18,file="aux.out")
do i=1,10
open(unit=101+i,file="fort.1+i")   !old file; unit and file name just
for simplicity

open(unit=1001+i,file="fort.101+i")  !New file

do j=1,EN  ! number of lines wrote to the new file
read(101+i,format)n1,n2,n3,n4,n5,n6,n7
 write(1001+i,format)n1,n2,n3,n4,n5,n6,n7

end o

do j=EN+1,N  ! number of lines left,N=total lines of old file
read(101+i,format)n1,n2,n3,n4,n5,n6,n7
 write(1001+i,format)n1,n2,n3,n4,n5,n6,n7

end do

close(101+i)
close(1001+i)

end do

news.gatech.edu <liscon@gmail.com> wrote:

>   When I use fortran 90 code to read larger files,about 1G each, and
> write the data to other opened files with 32-bit Intel-compiler. Each
> time, I just opened 3 files simultaneously and closed unused ones. The
> code always stops at the point of 10 files reading. I am not sure it is
> the compiler problem about the limitation of buffer memory.

[sample code elided. I don't think it is telling me much about the
problem. I saw some "issues" in the sample code, but I'd guess they were
all artifacts of the attempt at simplification and have nothing to do
with the question. For example, the file names were... well, not what
one might think they were intended to be.]

Not a lot of data to go on, but I doubt it has anthing to do with
"buffer memory." Compilers don't need to buffer the whole file or
anything like that. Large records might cause buffer size "issues", but
I don't see any suggetsion that your record sizes are particularly
large.

My first guess would be that the compiler and/or operating system (you
didn't say what operating system that I noticed; the Intel compiler runs
on several) don't support files as large as you are trying to write.

My second guess would be a memory leak somewhere. If so, simplified
illustrative code would not be enough to find it. The old MS
Powerstation 4 compiler used to have memory leaks in formatted I/O. I
wouldn't expect that out of Intel, but I can't rule it out.

P.S. You mention files of gigabyte size, and your example shows
formatted I/O. I assume you know that formatted I/O is *VERY*
inefficient, both in terms of file size and time. Maybe you have good
reason to use formatted I/O anyway; maybe you don't even have a choice.
And maybe performance is not an issue. But I thought I'd mention it just
in case.

--
Richard Maine                     | Good judgment comes from experience;
email: my first.last at org.domain| experience comes from bad judgment.
org: nasa, domain: gov            |       -- Mark Twain

I am reading your post as meaning you open and close files one after
another and the program stops after doing this 10 times.

The MS operating system allows you to execute in a DOS environment and
in a Windows console environment (and others) but these two can both be
used for a DOS-target compilation. Both have a limit on the number of
files being open at the same time. 5 files are already taken internally
but are in the count of active data control blocks. IF and only IF this
is what you are doing you have to change the number of DCB file blocks
to be assigned ata program start. This is usually done in autoexec.nt
for NT and XP systems (in \WINNT\System32. or similar folder) and in
autoexec.bat in the disc root (C:\) in earlier Windows systems.

I had this problem with a general sort-merge routine where I sometimes
needed 32 work files. I had to write an NT-specific native Windows
program because NT comes with a fixed limit for DOS emulations.

I don't thinkyou have a file size problem, without more information
Terence Wright

Terry <tbwright@cantv.net> wrote:

> IF and only IF this
> is what you are doing you have to change the number of DCB file blocks
> to be assigned ata program start.

The OP specifically mentioned f90. Do there exist any f90 compilers that
even can use old DOS DCB file blocks? I guess I don't know for sure that
there aren't, but my money goes on the side that says the problem lies
elsewhere. It has sure been a long time since I've had to pay any
attention to DCB file blocks. I've done it, but it has been a long time.
I sure don't recall it ever coming up in f90 compilers.

--
Richard Maine                    | Good judgement comes from experience;
email: last name at domain . net | experience comes from bad judgement.
domain: summertriangle           |  -- Mark Twain

Terry wrote:

> I am reading your post as meaning you open and close files one after
> another and the program stops after doing this 10 times.

> The MS operating system allows you to execute in a DOS environment and
> in a Windows console environment (and others) but these two can both be
> used for a DOS-target compilation. Both have a limit on the number of
> files being open at the same time. 5 files are already taken internally
> but are in the count of active data control blocks. IF and only IF this
> is what you are doing you have to change the number of DCB file blocks
> to be assigned ata program start.

If I remember for DOS there are DCB and FCBS parameters, which specify
the DCBs used by DOS 1.x programs, and FCBS used by programs for later
versions of DOS.  Hopefully most compilers available now use newer
methods than DCB, which doesn't allow for a path including subdirectories.

-- glen

Liscom:  Your program looks ok; others made suggestions on Formatted I/O
but this won't likely change the problem you're having. Suggest that you
check the following:

1) Check how much RAM and correspoinding virtual storage you have.
Many people suggest that virtual storage shoud be about 2.5 times RAM.
If that number is less than 3GB then you're suspicions are likely true.
(Note the Windows Control Panel System icon will show RAM size; typically
for the System Icon, the Performance button of the Advancde tab will show
Virtual Storage allocation. If you are running under Linux try the
following commands:
free -m
cat /proc/cpuinfo > t.t
cat /proc/meminfo > t.t

2) Make sure you're using the latest compiler version.

3) Try a different compiler; for example, G95: http://g95.sourceforge.net/

Good luck with it.

Skip Knoble

On Thu, 13 Jul 2006 17:20:20 -0400, "news.gatech.edu" <liscon@gmail.com> wro
te:

-|
-|Hello Pals:
-|
-|  When I use fortran 90 code to read larger files,about 1G each, and
-|write the data to other opened files with 32-bit Intel-compiler. Each
-|time, I just opened 3 files simultaneously and closed unused ones. The
-|code always stops at the point of 10 files reading. I am not sure it is
-|the compiler problem about the limitation of buffer memory.
-|
-|Thanks very much
-|
-|Liscon
-|
-|
-|Simple code:
-|
-|       open(unit=18,file="aux.out")
-|       do i=1,10
-|	open(unit=101+i,file="fort.1+i")   !old file; unit and file name just
-|for simplicity
-|
-|	open(unit=1001+i,file="fort.101+i")  !New file
-|
-|	   do j=1,EN  ! number of lines wrote to the new file
-|	      read(101+i,format)n1,n2,n3,n4,n5,n6,n7
-|	       write(1001+i,format)n1,n2,n3,n4,n5,n6,n7

-|	   end o
-|
-|	   do j=EN+1,N  ! number of lines left,N=total lines of old file
-|	      read(101+i,format)n1,n2,n3,n4,n5,n6,n7
-|	       write(1001+i,format)n1,n2,n3,n4,n5,n6,n7

-|	   end do
-|
-|	close(101+i)
-|	close(1001+i)
-|
-|       end do

Thanks very much!
To clarify the problem:

Memory leak: I did use some allocatable matrice and forget to deallocate
them, but I deallocate them or just declare them as static after you
mentioned. The same problem happned. so maybe memory leak is not
possible. The only parameters used often in the code are 7 integer
variables.

Compilers: I use Intel Compiler(version 8.1) on Linux.

Also the code runs smoothly for 64 files when the files' size are small,
say 100M each one. And when I increase the file size to about 400M each,
it stops at the point of 32 files. Moreover, when the file size
increases to about 1G, it just handles the first 10 files.

It always stops about 800 sec.

And the error output are:
Exited with exit code 1.

Resource usage summary:

CPU time   :    832.00 sec.
Max Memory :         2 MB
Max Swap   :        19 MB

Max Processes  :         1
Max Threads    :         1

The output (if any) follows:

Timeout alarm signaled

TID  HOST_NAME    COMMAND_LINE            STATUS            TERMINATION_TIME
==== ========== ================  =======================
===================
0001 compute-6- mpirun_ssh -np 1  Killed by PAM (SIGTERM)  07/13/2006
23:08:10


About the Formatted and Unformatted I/O, I don't know much about the
efficiency. Thanks very much, I will try it right now.

Liscon


Richard E Maine wrote:

> news.gatech.edu <liscon@gmail.com> wrote:
>
> 
>
>
> [sample code elided. I don't think it is telling me much about the
> problem. I saw some "issues" in the sample code, but I'd guess they were
> all artifacts of the attempt at simplification and have nothing to do
> with the question. For example, the file names were... well, not what
> one might think they were intended to be.]
>
> Not a lot of data to go on, but I doubt it has anthing to do with
> "buffer memory." Compilers don't need to buffer the whole file or
> anything like that. Large records might cause buffer size "issues", but
> I don't see any suggetsion that your record sizes are particularly
> large.
>
> My first guess would be that the compiler and/or operating system (you
> didn't say what operating system that I noticed; the Intel compiler runs
> on several) don't support files as large as you are trying to write.
>
> My second guess would be a memory leak somewhere. If so, simplified
> illustrative code would not be enough to find it. The old MS
> Powerstation 4 compiler used to have memory leaks in formatted I/O. I
> wouldn't expect that out of Intel, but I can't rule it out.
>
> P.S. You mention files of gigabyte size, and your example shows
> formatted I/O. I assume you know that formatted I/O is *VERY*
> inefficient, both in terms of file size and time. Maybe you have good
> reason to use formatted I/O anyway; maybe you don't even have a choice.
> And maybe performance is not an issue. But I thought I'd mention it just
> in case.
>

Thanks for new solutions.

For I am using Linux system, is there still DCB file block problem? If
so, how can change the parameters?

As Herman mentioned, I use the cammand "free -m" and get a output file
t.t with the information:

total:    used:    free:  shared: buffers:  cached:
Mem:  6238453760 6204731392 33722368        0 140128256 5121093632
Swap: 2147475456  1241088 2146234368
MemTotal:      6092240 kB
MemFree:         32932 kB
MemShared:           0 kB
Buffers:        136844 kB
Cached:        5000004 kB
SwapCached:       1064 kB
Active:        3127256 kB
ActiveAnon:     702984 kB
ActiveCache:   2424272 kB
Inact_dirty:   2085100 kB
Inact_laundry:  474264 kB
Inact_clean:    151596 kB
Inact_target:  1167640 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      6092240 kB
LowFree:         32932 kB
SwapTotal:     2097144 kB
SwapFree:      2095932 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

So which parameter should and how can I change it?


Thanks all you guys
Liscon





Herman D. Knoble wrote:

> Liscom:  Your program looks ok; others made suggestions on Formatted I/O
> but this won't likely change the problem you're having. Suggest that you
> check the following:
>
> 1) Check how much RAM and correspoinding virtual storage you have.
> Many people suggest that virtual storage shoud be about 2.5 times RAM.
> If that number is less than 3GB then you're suspicions are likely true.
> (Note the Windows Control Panel System icon will show RAM size; typically
> for the System Icon, the Performance button of the Advancde tab will show
> Virtual Storage allocation. If you are running under Linux try the
> following commands:
> free -m
> cat /proc/cpuinfo > t.t
> cat /proc/meminfo > t.t
>
> 2) Make sure you're using the latest compiler version.
>
> 3) Try a different compiler; for example, G95: http://g95.sourceforge.net/
>
> Good luck with it.
>
> Skip Knoble
>
> On Thu, 13 Jul 2006 17:20:20 -0400, "news.gatech.edu" <liscon@gmail.com> w
rote:
>
> -|
> -|Hello Pals:
> -|
> -|  When I use fortran 90 code to read larger files,about 1G each, and
> -|write the data to other opened files with 32-bit Intel-compiler. Each
> -|time, I just opened 3 files simultaneously and closed unused ones. The
> -|code always stops at the point of 10 files reading. I am not sure it is
> -|the compiler problem about the limitation of buffer memory.
> -|
> -|Thanks very much
> -|
> -|Liscon
> -|
> -|
> -|Simple code:
> -|
> -|       open(unit=18,file="aux.out")
> -|       do i=1,10
> -|	open(unit=101+i,file="fort.1+i")   !old file; unit and file name just
> -|for simplicity
> -|
> -|	open(unit=1001+i,file="fort.101+i")  !New file
> -|
> -|	   do j=1,EN  ! number of lines wrote to the new file
> -|	      read(101+i,format)n1,n2,n3,n4,n5,n6,n7
> -|	       write(1001+i,format)n1,n2,n3,n4,n5,n6,n7

> -|	   end o
> -|
> -|	   do j=EN+1,N  ! number of lines left,N=total lines of old file
> -|	      read(101+i,format)n1,n2,n3,n4,n5,n6,n7
> -|	       write(1001+i,format)n1,n2,n3,n4,n5,n6,n7

> -|	   end do
> -|
> -|	close(101+i)
> -|	close(1001+i)
> -|
> -|       end do
>

news.gatech.edu <liscon@gmail.com> wrote:


> For I am using Linux system, is there still DCB file block problem?

No. That one I know for sure. DCBs are purely a DOS thing - ancient
versions of DOS at that. I don't think there are even DCB issues with
with any f90 compilers on Windows, but that part I could be wrong on. No
way that DCBs have anything to do with Linux.

I'm afraid I don't have any further constructive suggestions other than
the possibilities I mentioned before. Your numbers from another posting
all sure seem to indicate that you stop at about the same total size. I
have a hard time imagining that this is a coincidence. It would, of
course, also correspond to processing about the same total number of
records, and thus, for example, the same number of repetitions of
anything that might leak memory. Many things could be correlated with
the file size, but your numbers pretty convincingly support my thesis
that it is likely not related to the number of files. I didn't think it
was anyway, so maybe I'm just seeing my prior prejudices in the data -
but I don't think so.

(It should be "obvious" and I'd guess it was not your problem, but I've
seen it misssed before, so just in case... you aren't running out of
disk space are you?)

--
Richard Maine                     | Good judgment comes from experience;
email: my first.last at org.domain| experience comes from bad judgment.
org: nasa, domain: gov            |       -- Mark Twain

Liscon:  It looks like you have plenty of memory for buffers.

Can you upgrade to Intel Compiler v9.1?
Or can you try G95 for Linux?
Or try the 64-bit Intel compiler? (If you have integer counters,
and you ar counting past the 2GB line (2**31-1) then you would
need to declare such integer counters Integer*8)

Skip


On Fri, 14 Jul 2006 10:36:04 -0400, "news.gatech.edu" <liscon@gmail.com> wro
te:

-|
-|
-|
-|Thanks for new solutions.
-|
-|For I am using Linux system, is there still DCB file block problem? If
-|so, how can change the parameters?
-|
-|As Herman mentioned, I use the cammand "free -m" and get a output file
-|t.t with the information:
-|
-|total:    used:    free:  shared: buffers:  cached:
-|Mem:  6238453760 6204731392 33722368        0 140128256 5121093632
-|Swap: 2147475456  1241088 2146234368
-|MemTotal:      6092240 kB
-|MemFree:         32932 kB
-|MemShared:           0 kB
-|Buffers:        136844 kB
-|Cached:        5000004 kB
-|SwapCached:       1064 kB
-|Active:        3127256 kB
-|ActiveAnon:     702984 kB
-|ActiveCache:   2424272 kB
-|Inact_dirty:   2085100 kB
-|Inact_laundry:  474264 kB
-|Inact_clean:    151596 kB
-|Inact_target:  1167640 kB
-|HighTotal:           0 kB
-|HighFree:            0 kB
-|LowTotal:      6092240 kB
-|LowFree:         32932 kB
-|SwapTotal:     2097144 kB
-|SwapFree:      2095932 kB
-|HugePages_Total:     0
-|HugePages_Free:      0
-|Hugepagesize:     2048 kB
-|
-|So which parameter should and how can I change it?
-|
-|
-|Thanks all you guys
-|Liscon
-|
-|
-|
-|
-|
-|Herman D. Knoble wrote:
-|
-|> Liscom:  Your program looks ok; others made suggestions on Formatted I/O
-|> but this won't likely change the problem you're having. Suggest that you
-|> check the following:
-|>
-|> 1) Check how much RAM and correspoinding virtual storage you have.
-|> Many people suggest that virtual storage shoud be about 2.5 times RAM.
-|> If that number is less than 3GB then you're suspicions are likely true.
-|> (Note the Windows Control Panel System icon will show RAM size; typicall
y
-|> for the System Icon, the Performance button of the Advancde tab will sho
w
-|> Virtual Storage allocation. If you are running under Linux try the
-|> following commands:
-|> free -m
-|> cat /proc/cpuinfo > t.t
-|> cat /proc/meminfo > t.t
-|>
-|> 2) Make sure you're using the latest compiler version.
-|>
-|> 3) Try a different compiler; for example, G95: [url]http://g95.sourceforge.net/[/ur
l]
-|>
-|> Good luck with it.
-|>
-|> Skip Knoble
-|>
-|> On Thu, 13 Jul 2006 17:20:20 -0400, "news.gatech.edu" <liscon@gmail.com>
 wrote:
-|>
-|> -|
-|> -|Hello Pals:
-|> -|
-|> -|  When I use fortran 90 code to read larger files,about 1G each, and
-|> -|write the data to other opened files with 32-bit Intel-compiler. Each
-|> -|time, I just opened 3 files simultaneously and closed unused ones. The
-|> -|code always stops at the point of 10 files reading. I am not sure it i
s
-|> -|the compiler problem about the limitation of buffer memory.
-|> -|
-|> -|Thanks very much
-|> -|
-|> -|Liscon
-|> -|
-|> -|
-|> -|Simple code:
-|> -|
-|> -|       open(unit=18,file="aux.out")
-|> -|       do i=1,10
-|> -|	open(unit=101+i,file="fort.1+i")   !old file; unit and file name just
-|> -|for simplicity
-|> -|
-|> -|	open(unit=1001+i,file="fort.101+i")  !New file
-|> -|
-|> -|	   do j=1,EN  ! number of lines wrote to the new file
-|> -|	      read(101+i,format)n1,n2,n3,n4,n5,n6,n7
-|> -|	       write(1001+i,format)n1,n2,n3,n4,n5,n6,n7

-|> -|	   end o
-|> -|
-|> -|	   do j=EN+1,N  ! number of lines left,N=total lines of old file
-|> -|	      read(101+i,format)n1,n2,n3,n4,n5,n6,n7
-|> -|	       write(1001+i,format)n1,n2,n3,n4,n5,n6,n7

-|> -|	   end do
-|> -|
-|> -|	close(101+i)
-|> -|	close(1001+i)
-|> -|
-|> -|       end do
-|>