Visual Basic - large data in memory

Author

large data in memory

2006-02-27, 6:56 pm

Hello,

We are building a little application to do a large amount of data
manipulation. We will query from a database a large amount of rows. From
that will pull an id out. From each id we will query from another table
more information. We were thinking of dimming an array when we know how
many ids we have. Then for the array have an associated dictionary for each
id.

We might end up with 10,000 to 100,000 array entries. And the associated
dictionary that would contain a few values. The id is only a 27 char
string. And the other values are strings only 10 to 20 chars.

Does this seem like a large amount of data to keep in storage? To
manipulate?

Or do people code stuff like this all the time?

Thanks for any help

R. MacDonald

2006-02-28, 3:55 am

Hello, bp,

Re:
> Or do people code stuff like this all the time?

Yes (unfortunately). ;-)

But "Large" is a relative term, and it's a difficult to be definitive
without knowing precisely what you are trying to do. It sounds like you
will be making inefficient use of a significant amount of space. But
there are circumstances where this could still be a reasonable thing to
do. Questions like:

How often will the application be used?
How quickly must the data be accessed?
How much time can we devote to development?

can all influence this. For example, for a "little application" that
will only be used once, the approach mentioned might be quite
reasonable. (However, watch out for those applications that will only
be used "once". They have a tendency to get reused, recycled, enhanced,
and expanded until they become monsters.)

If you need to, you can find several ways to improve efficiency. For
example (depending on the type of "database" you are using) you may
find that you can access only the records actually required with a
single query by using an "inner join". If you are using only a few
elements from a very large array, a "hash table" might provide a better
storage solution.

Cheers,
Randy

bp wrote:
> Hello,
>
> We are building a little application to do a large amount of data
> manipulation. We will query from a database a large amount of rows. From
> that will pull an id out. From each id we will query from another table
> more information. We were thinking of dimming an array when we know how
> many ids we have. Then for the array have an associated dictionary for each
> id.
>
> We might end up with 10,000 to 100,000 array entries. And the associated
> dictionary that would contain a few values. The id is only a 27 char
> string. And the other values are strings only 10 to 20 chars.
>
> Does this seem like a large amount of data to keep in storage? To
> manipulate?
>
> Or do people code stuff like this all the time?
>
>
>
> Thanks for any help
>
>
>
>

Sponsored Links