VC STL - Large file support in iostream library

Author

Large file support in iostream library

Etienne Fortin

2005-03-29, 4:02 pm

I'm currently writing code that works with file bigger than 2 gigs. I assumed
that the iostream library support 64-bits file pointer by default, and to my
surprise it doesn't!!! Am I forgetting to #define something? Is there a
switch to set at the command line?

I'm a little surprised that the iostream library, even on the latest of the
Visual C++ compiler, doesn't support large file by default.

Etienne Fortin

Carl Daniel [VC++ MVP]

2005-03-29, 4:02 pm

"Etienne Fortin" <EtienneFortin@discussions.microsoft.com> wrote in message
news:F1A1B123-89E4-4993-B678-59253B4EF214@microsoft.com...
> I'm currently writing code that works with file bigger than 2 gigs. I
> assumed
> that the iostream library support 64-bits file pointer by default, and to
> my
> surprise it doesn't!!! Am I forgetting to #define something? Is there a
> switch to set at the command line?

Nope - it's just not supported.

> I'm a little surprised that the iostream library, even on the latest of
> the
> Visual C++ compiler, doesn't support large file by default.

It is a little bit surprising. Then again, C++ streams are so grotequely
inefficient, I really can't imagine a program accessing a file of >4Gb using
them. You'll have to use the Win32 file API for large file support (IIRC,
the C file I/O doesn't support 64 bit file offsets either).

-cd

Bo Persson

2005-03-29, 9:01 pm

"Carl Daniel [VC++ MVP]"
<cpdaniel_remove_this_and_nospam@mvps.org.nospam> skrev i meddelandet
news:OH%234CdINFHA.2136@TK2MSFTNGP14.phx.gbl...
> "Etienne Fortin" <EtienneFortin@discussions.microsoft.com> wrote in
> message news:F1A1B123-89E4-4993-B678-59253B4EF214@microsoft.com...
>
> Nope - it's just not supported.
>
>
> It is a little bit surprising. Then again, C++ streams are so
> grotequely inefficient, I really can't imagine a program accessing a
> file of >4Gb using them. You'll have to use the Win32 file API for
> large file support (IIRC, the C file I/O doesn't support 64 bit file
> offsets either).

It can, if there is any integer type wide enough.

C++ streams are limited on implementations having a 32 bit 'long'. It's
not a language thing as such.

Bo Persson

Carl Daniel [VC++ MVP]

2005-03-29, 9:01 pm

"Bo Persson" <bop@gmb.dk> wrote in message
news:ONmfzPKNFHA.2580@TK2MSFTNGP09.phx.gbl...
> "Carl Daniel [VC++ MVP]"
>
> It can, if there is any integer type wide enough.
>
> C++ streams are limited on implementations having a 32 bit 'long'. It's
> not a language thing as such.

No, of course it's not. I merely meant that the VC++ implementation of C
file I/O also doesn't support 64 bit file offsets (and since the C++
IOStreams implementation is built atop the C file I/O, it's no surprise that
C++ IOStreams doesn't support them either).

-cd

Etienne Fortin

2005-03-30, 4:02 pm

> > I'm a little surprised that the iostream library, even on the latest of
>
> It is a little bit surprising. Then again, C++ streams are so grotequely
> inefficient, I really can't imagine a program accessing a file of >4Gb using
> them. You'll have to use the Win32 file API for large file support (IIRC,
> the C file I/O doesn't support 64 bit file offsets either).
>

Are they that inefficient? Are they less efficient than, let's say, streams
in .NET?

Etienne Fortin

Etienne Fortin

2005-03-30, 4:02 pm

> No, of course it's not. I merely meant that the VC++ implementation of C
> file I/O also doesn't support 64 bit file offsets (and since the C++
> IOStreams implementation is built atop the C file I/O, it's no surprise that
> C++ IOStreams doesn't support them either).
>

It seems that if MAX_INTEGRAL_something is defined to 64 or more, all the
xxxi64 C I/O functions are also defined. And if _WIN64 is defined, then the
C++ iostream uses 64 bits file pointers and position. If you can do it with
_WIN64, then it should be as easy to do it on a 32-bits computer... There
should be an option to use the iostream with 64-bits file access. Everything
is already there.

Etienne

Carl Daniel [VC++ MVP]

2005-03-30, 4:02 pm

Etienne Fortin wrote:
>
> Are they that inefficient? Are they less efficient than, let's say,
> streams in .NET?
>

I don't know - I wouldn't think so.

In my experience, programs that deal with very large files are usually
dealing with one of two things: 1. large databases, 2. large media files.
In either case, they're not really stream oriented files, but rather block
oriented files that are better accessed using platform specific mechanisms
that standard C or C++ file I/O.

Of course, your application may vary. I can't really imagine having a text
file of over 4Gb in size, but no doubt applications exist that create and
process such large files in a "streamy" way.

-cd

Tom Widmer

2005-03-30, 4:02 pm

Etienne Fortin wrote:
>
>
> It seems that if MAX_INTEGRAL_something is defined to 64 or more, all the
> xxxi64 C I/O functions are also defined. And if _WIN64 is defined, then the
> C++ iostream uses 64 bits file pointers and position. If you can do it with
> _WIN64, then it should be as easy to do it on a 32-bits computer... There
> should be an option to use the iostream with 64-bits file access. Everything
> is already there.

Yes, 64-bit file access is possible in the C++ standard even on a 16-bit
platform I think - that's what fpos and streampos are for - fpos could
encapsulate an arbitrary number of bits of file position information if
the OS supports it.

Tom

Gene Bushuyev

2005-04-01, 4:02 am

"Carl Daniel [VC++ MVP]" <cpdaniel_remove_this_and_nospam@mvps.org.nospam>
wrote in message news:eDLc4VTNFHA.244@tk2msftngp13.phx.gbl...
>
> In my experience, programs that deal with very large files are usually
> dealing with one of two things: 1. large databases, 2. large media files.
> In either case, they're not really stream oriented files, but rather block
> oriented files that are better accessed using platform specific mechanisms
> that standard C or C++ file I/O.

I beg to differ. There are many EDA/CAD applications that produce and
consume big text files in a number of industry standard formats, where
10-100GB files are typical for medium-to-large size tasks. And 64-bit
implementations of standard streams on a number of platforms are working
just fine. There is no problem with stream performance in these applications
as syntax parsing and semantic actions taking more time than formatted
reading and writing. Those applications also often use compression schemes,
which add additional overhead, but which also are easy to implement on top
of standard basic_streambuf. Moreover, for the last decade the
processor/memory performance grew faster than I/O speed, making formatted
in/output less and less of an overhead. Having the convenience, flexibility,
portability, and safety of standard streams by far outweighs slightly better
performance of platform-specific solutions.

Carl Nettelblad

2005-04-22, 9:00 am

> I don't know - I wouldn't think so.
>
> In my experience, programs that deal with very large files are usually
> dealing with one of two things: 1. large databases, 2. large media files.
> In either case, they're not really stream oriented files, but rather block
> oriented files that are better accessed using platform specific mechanisms
> that standard C or C++ file I/O.
Well, I don't see why a call to fread would be much slower than a call to
ReadFile, unless you really make use of the additional featues of ReadFile
(like overlapped I/O, for a start).

I wouldn't use fgetc or fscanf in a block I/O context, of course, but why
would I?

/Carl

Carl Daniel [VC++ MVP]

2005-04-22, 4:04 pm

Carl Nettelblad wrote:
[color=darkred]
> Well, I don't see why a call to fread would be much slower than a
> call to ReadFile, unless you really make use of the additional
> featues of ReadFile (like overlapped I/O, for a start).

fread is not a C++ iostreams function, which is what was being discussed.
fread can be significantly less efficient though if the file was opened in
'text mode' (the default). In text mode, fread will scan the data looking
for \r\n sequences which will be replaced with \n, neither ReadFile, nor
fread on a file opened in binary mode will do that.

-cd

Stephen Howe

2005-04-25, 4:02 pm

> Of course, your application may vary. I can't really imagine having a
text
> file of over 4Gb in size...

We are getting there. We receive text files that contain TV rating data. For
some countries with the proliferation of new channels, the text files
containing audience, programme and commercial data get bigger with each
passing day.

Stephen Howe

Sponsored Links