Home > Archive > Visual Basic > January 2006 > Opening large files for small amounts of data
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Opening large files for small amounts of data
|
|
|
| When I open a file in VB, what really happens? What I mean is that I
may have a file on a remote server, but still inside of our network,
that can be over 1 GB in size. If I just need to open it to append a
small amount of data to it, how much traffic over the network really
occurs?
How about for reading a couple of lines from that file (say the last 2
lines)?
Are there things I can do to make sure that I can get the fastest
access times?
How does a computer actually "Open" a file, and how much data is sent
at that time?
I am so ignorant in this area that I'm sure I'm asking stupid
questions, so if anyone can point me to some information that would be
great. I didn't have that much luck when searching.
Matt
| |
| Randy Birch 2006-01-09, 7:16 pm |
| : When I open a file in VB, what really happens? What I mean is that I
: may have a file on a remote server, but still inside of our network,
: that can be over 1 GB in size. If I just need to open it to append a
: small amount of data to it, how much traffic over the network really
: occurs?
AFAIK, it just acquires access to the file with no significant traffic. It
doesn't haul down the file, for example.
: How about for reading a couple of lines from that file (say the last 2
: lines)?
If you're opening in Binary mode and using S to move the file pointer to
a specific spot, or using Random Access mode with fixed-size UDTs, you can
access any point or record in the file in random order, forwards or
backwards. If the file is opened for Append, however, the only way to move
to the last couple of lines for reading is to read each line starting from
the beginning. You could, I suppose, try mixing a S call after the file
was opened for append but you'd have to handle tracking where the data you
want to read is located.
: Are there things I can do to make sure that I can get the fastest
: access times?
All are fast for access; any sequential file method (input/output/append)
will require reading from the start of the file to reach any point later in
the file (for reading). Naturally append itself automatically expects to add
new data to the end. So if you can bypass any reading of the file's current
data, you're overhead is just that needed to pass the data to be written to
the file.
: How does a computer actually "Open" a file, and how much data is sent
: at that time?
It acquires a handle to the file from the operating system. Probably no more
than a few bytes are involved in the actual work in doing this. The largest
overhead is hauling the file data down to the client system (i.e. a file
read), or re-writing an entire file at the destination (re-writing from
memory, as opposed to copying the file itself).
--
Randy Birch
MS MVP Visual Basic
http://vbnet.mvps.org/
----------------------------------------------------------------------------
Read. Decide. Sign the petition to Microsoft.
http://classicvb.org/petition/
----------------------------------------------------------------------------
| |
|
| Randy, thank you SOOOO much for that. I consider myself a pretty good
programmer (mostly thanks to this group, your site, and
vbaccelerator.com), but there are huge holes in my knowledge since I
was a Political Science major.
> If you're opening in Binary mode and using S to move the file pointer to
> a specific spot, or using Random Access mode with fixed-size UDTs, you can
> access any point or record in the file in random order, forwards or
> backwards.
So this is how MS Access works, I think. When you open up an Access DB
somewhere on the network, it doesn't take as long to open it as it
would to copy the whole file down locally and then open it. It just
knows where to find things like the table listing and just ss to
find that. And when you add data to a table, the file grows in chunks,
not just by the exact amount of data. I'm assuming so that it can use
S to write data to already pre-blocked off areas of the file.
Matt
| |
|
| Okay, I replied to this about 5 hours ago thanking you. In case it
doesn't show up, hopefully this will -- thanks so much -- that was
invaluable.
Matt
| |
|
|
"YYZ" <matt.dawdy@gmail.com> wrote in message
news:1135699468.614195.305160@g47g2000cwa.googlegroups.com...
> So this is how MS Access works, I think. When you open up an Access DB
> somewhere on the network, it doesn't take as long to open it as it
> would to copy the whole file down locally and then open it. It just
> knows where to find things like the table listing and just ss to
> find that. And when you add data to a table, the file grows in chunks,
> not just by the exact amount of data. I'm assuming so that it can use
> S to write data to already pre-blocked off areas of the file.
Well, that's probably way, WAY over-simplified, if even remotely close. <g>
I make no pretension of knowing anything at all about the internal structure
of an .MDB file. But, an Access database has a number of system tables that
are normally hidden. Some of these system tables contain information about
the structure of the database (tables, columns, indexes, keys,
relationships, etc.). It's the Jet engine that knows how to look up things
in these system tables and in non-system tables. IOW, it doesn't really have
much to do with file I/O. It's more the Jet engine.
--
Mike
Microsoft MVP Visual Basic
| |
|
| > Well, that's probably way, WAY over-simplified, if even remotely close. <g>
Right -- I didn't mean to imply that I had cracked Jet's method of
database serving by any means. All I meant was that I originally
thought that to read anything out of a file that the entire file had to
be copied down remotely. Jet obviously doesn't do that since I can
open up an 800 MB database in a couple of seconds, yet to copy it down
to my local machine can take 15-20 minutes or so, so obviously I was
not understanding something, hence my original question.
Also, I didn't distinguish between Jet and Access. I really do know
the difference there. Sorry I kinda lumped those together. I was
speaking VERY simplistically, and should have said so.
Matt
|
|
|
|
|