Home > Archive > Visual Basic > August 2005 > Copying Similar Data Structures
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Copying Similar Data Structures
|
|
| Tim Rude 2005-08-29, 9:55 pm |
| I've got a complex Type ... End Type data structure consisting of
fixed-length strings, booleans, dates, integers, and some arrays of the
same - all together each 'record' of this type is exactly 2030 bytes (there
are no variable length strings in the type). Something like this:
Type MyData
Value1 As String * 20
Value2 As Integer
Value3 As Boolean
Value4 As Date
...
W s(1 To 53) As Date
WOK(1 To 53) As Boolean
Something(1 To 30, 1 To 2) As Date
End Type
I use the Put/Get commands for reading and writing this to a random access
file on disk.
Now I need to add one more item to the end of the Type, a single Byte value.
This pushes my 'record' length to 2031. i.e.:
Type MyData
Value1 As String * 20
Value2 As Integer
Value3 As Boolean
Value4 As Date
...
Ws(1 To 53) As Date
WOK(1 To 53) As Boolean
Something(1 To 30, 1 To 2) As Date
NewValue As Byte
End Type
What would be the quickest / easiest way to read the old records (2030
bytes) from the random access file and convert them into an array of the new
type structure (2031 bytes)?
Obviously, I could read the old records one by one and transfer the data one
field at a time into the new record structure. But there's gotta be a better
way, right?
--
Tim Rude
timrude@NOSPAM.hotmail.com
(remove NOSPAM. for correct email address)
| |
|
| I think you are going to be stuck reading the records of the old file format
and convert them to the new structure / file format...
--
Chris Hanscom - Microsoft MVP (VB)
Veign's Resource Center
http://www.veign.com/vrc_main.asp
--
"Tim Rude" <timrude@nospam.hotmail.com> wrote in message
news:Os6oKUQrFHA.3216@TK2MSFTNGP12.phx.gbl...
> I've got a complex Type ... End Type data structure consisting of
> fixed-length strings, booleans, dates, integers, and some arrays of the
> same - all together each 'record' of this type is exactly 2030 bytes
(there
> are no variable length strings in the type). Something like this:
>
> Type MyData
> Value1 As String * 20
> Value2 As Integer
> Value3 As Boolean
> Value4 As Date
> ...
> Ws(1 To 53) As Date
> WOK(1 To 53) As Boolean
> Something(1 To 30, 1 To 2) As Date
> End Type
>
> I use the Put/Get commands for reading and writing this to a random access
> file on disk.
>
> Now I need to add one more item to the end of the Type, a single Byte
value.
> This pushes my 'record' length to 2031. i.e.:
>
> Type MyData
> Value1 As String * 20
> Value2 As Integer
> Value3 As Boolean
> Value4 As Date
> ...
> Ws(1 To 53) As Date
> WOK(1 To 53) As Boolean
> Something(1 To 30, 1 To 2) As Date
> NewValue As Byte
> End Type
>
> What would be the quickest / easiest way to read the old records (2030
> bytes) from the random access file and convert them into an array of the
new
> type structure (2031 bytes)?
>
> Obviously, I could read the old records one by one and transfer the data
one
> field at a time into the new record structure. But there's gotta be a
better
> way, right?
>
> --
> Tim Rude
>
> timrude@NOSPAM.hotmail.com
> (remove NOSPAM. for correct email address)
>
>
| |
| Tim Rude 2005-08-29, 9:55 pm |
| After posting the question, I had a go with using the CopyMemory API and it
seemed to work.
Assuming my two UDT structures are identical except for the additional
item(s) added to the end of the larger structure, the following seems to
work. Am I deluding myself?
=======
Public Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" _
(Destination As Any, Source As Any, ByVal Length As Long)
Public Type UDT1
...
End Type
Public Type UDT2
... 'same lines as UDT1
NewData as Byte 'new item added at very end of UDT
End Type
Public Array1() as UDT1
Public Array2() as UDT2
Public Sub Test()
Redim Preserve Array1(5)
Redim Preserve Array2(5)
Array1(3).SomeValue = ...
CopyMemory Array2(3), Array1(3), Len(Array1(3))
Debug.Print Array2(3).SomeValue
End Sub
--
Tim Rude
timrude@NOSPAM.hotmail.com
(remove NOSPAM. for correct email address)
"Veign" <NOSPAMinveign@veign.com> wrote in message
news:%23ABDqnQrFHA.1724@TK2MSFTNGP14.phx.gbl...
> I think you are going to be stuck reading the records of the old file
format
> and convert them to the new structure / file format...
>
> --
> Chris Hanscom - Microsoft MVP (VB)
> Veign's Resource Center
> http://www.veign.com/vrc_main.asp
> --
>
>
> "Tim Rude" <timrude@nospam.hotmail.com> wrote in message
> news:Os6oKUQrFHA.3216@TK2MSFTNGP12.phx.gbl...
> (there
access[color=darkred]
> value.
> new
> one
> better
>
>
| |
| Jim Mack 2005-08-29, 9:55 pm |
| Tim Rude wrote:
> I've got a complex Type ... End Type data structure consisting of
> fixed-length strings, booleans, dates, integers, and some arrays of
> the same - all together each 'record' of this type is exactly 2030
> bytes (there are no variable length strings in the type). Something
> like this:=20
I can think of a couple of ways, but they're somewhat fanciful and =
without testing I couldn't say if they'd be any faster.
One would be to read the old file record by record into the new array, =
and adjust the file pointer after each get:
Get #old,,NewRecordType ' get an old record and an extra byte
S #old, S(#old) - 1 ' back up over the extra byte
loop
This would leave garbage in the new byte, if that matters.
The other way would be to use CopyMemory, incrementing the source and =
destination pointers by different amounts. Because you have a string =
value as the first field of the UDT, you'd have to read the SAFEARRAY =
headers to get the actual starting data addresses and increments. Not a =
huge deal, but not for the casual user either. =20
If you don't feel like probing SAFEARRAYs, you could try using =
(VarPtr(Old|NewArray(0).Value2) - 40) for the starting values, copy =
(LenB(OldArray(0)) bytes per loop, and increment by =
(LenB(Old|NewArray(0)) for each. That should work, but it needs testing.
--=20
Jim Mack
MicroDexterity Inc
www.microdexterity.com
> Type MyData
> Value1 As String * 20
> Value2 As Integer
> Value3 As Boolean
> Value4 As Date
> ...
> Ws(1 To 53) As Date
> WOK(1 To 53) As Boolean
> Something(1 To 30, 1 To 2) As Date
> End Type
>=20
> I use the Put/Get commands for reading and writing this to a random
> access file on disk.
>=20
> Now I need to add one more item to the end of the Type, a single Byte
> value. This pushes my 'record' length to 2031. i.e.:
>=20
> Type MyData
> Value1 As String * 20
> Value2 As Integer
> Value3 As Boolean
> Value4 As Date
> ...
> Ws(1 To 53) As Date
> WOK(1 To 53) As Boolean
> Something(1 To 30, 1 To 2) As Date
> NewValue As Byte
> End Type
>=20
> What would be the quickest / easiest way to read the old records (2030
> bytes) from the random access file and convert them into an array of
> the new type structure (2031 bytes)?
>=20
> Obviously, I could read the old records one by one and transfer the
> data one field at a time into the new record structure. But there's
> gotta be a better way, right?
| |
| Tim Rude 2005-08-30, 3:55 am |
| Jim,
Since all my strings are fixed-length, do I need the SAFEARRAY stuff?
I tried using a straight CopyMemory call and it seemed to work great.
i.e.: Read records from file into old structure one record at a time.
CopyMemory from old structure to new structure and then fill in any
necessary data into the new field items. Read next record from file, etc.
....
When finished I have array of new structure to write back to disk. Seems to
work. I can query various values from the array of the new structure and so
far it's been perfect. And no crash and burn yet...
Is this too simple?
--
Tim Rude
timrude@NOSPAM.hotmail.com
(remove NOSPAM. for correct email address)
"Jim Mack" <jmack@mdxi.nospam.com> wrote in message
news:%23Szx81QrFHA.1008@TK2MSFTNGP10.phx.gbl...
Tim Rude wrote:
> I've got a complex Type ... End Type data structure consisting of
> fixed-length strings, booleans, dates, integers, and some arrays of
> the same - all together each 'record' of this type is exactly 2030
> bytes (there are no variable length strings in the type). Something
> like this:
I can think of a couple of ways, but they're somewhat fanciful and without
testing I couldn't say if they'd be any faster.
One would be to read the old file record by record into the new array, and
adjust the file pointer after each get:
Get #old,,NewRecordType ' get an old record and an extra byte
S #old, S(#old) - 1 ' back up over the extra byte
loop
This would leave garbage in the new byte, if that matters.
The other way would be to use CopyMemory, incrementing the source and
destination pointers by different amounts. Because you have a string value
as the first field of the UDT, you'd have to read the SAFEARRAY headers to
get the actual starting data addresses and increments. Not a huge deal, but
not for the casual user either.
If you don't feel like probing SAFEARRAYs, you could try using
(VarPtr(Old|NewArray(0).Value2) - 40) for the starting values, copy
(LenB(OldArray(0)) bytes per loop, and increment by (LenB(Old|NewArray(0))
for each. That should work, but it needs testing.
--
Jim Mack
MicroDexterity Inc
www.microdexterity.com
> Type MyData
> Value1 As String * 20
> Value2 As Integer
> Value3 As Boolean
> Value4 As Date
> ...
> Ws(1 To 53) As Date
> WOK(1 To 53) As Boolean
> Something(1 To 30, 1 To 2) As Date
> End Type
>
> I use the Put/Get commands for reading and writing this to a random
> access file on disk.
>
> Now I need to add one more item to the end of the Type, a single Byte
> value. This pushes my 'record' length to 2031. i.e.:
>
> Type MyData
> Value1 As String * 20
> Value2 As Integer
> Value3 As Boolean
> Value4 As Date
> ...
> Ws(1 To 53) As Date
> WOK(1 To 53) As Boolean
> Something(1 To 30, 1 To 2) As Date
> NewValue As Byte
> End Type
>
> What would be the quickest / easiest way to read the old records (2030
> bytes) from the random access file and convert them into an array of
> the new type structure (2031 bytes)?
>
> Obviously, I could read the old records one by one and transfer the
> data one field at a time into the new record structure. But there's
> gotta be a better way, right?
| |
| NickHK 2005-08-30, 3:55 am |
| Tim,
Not sure it's any better than your CopyMem, but:
Public Type UDT1
...
End Type
Public Type UDT2
OldData as UDT1
NewData as Byte 'new item added at very end of UDT
End Type
NickHK
"Tim Rude" <timrude@nospam.hotmail.com> wrote in message
news:#9Aa9$QrFHA.1032@TK2MSFTNGP12.phx.gbl...
> Jim,
>
> Since all my strings are fixed-length, do I need the SAFEARRAY stuff?
>
> I tried using a straight CopyMemory call and it seemed to work great.
>
> i.e.: Read records from file into old structure one record at a time.
> CopyMemory from old structure to new structure and then fill in any
> necessary data into the new field items. Read next record from file, etc.
> ...
>
> When finished I have array of new structure to write back to disk. Seems
to
> work. I can query various values from the array of the new structure and
so
> far it's been perfect. And no crash and burn yet...
>
> Is this too simple?
>
> --
> Tim Rude
>
> timrude@NOSPAM.hotmail.com
> (remove NOSPAM. for correct email address)
>
> "Jim Mack" <jmack@mdxi.nospam.com> wrote in message
> news:%23Szx81QrFHA.1008@TK2MSFTNGP10.phx.gbl...
> Tim Rude wrote:
>
> I can think of a couple of ways, but they're somewhat fanciful and without
> testing I couldn't say if they'd be any faster.
>
> One would be to read the old file record by record into the new array, and
> adjust the file pointer after each get:
>
> Get #old,,NewRecordType ' get an old record and an extra byte
> S #old, S(#old) - 1 ' back up over the extra byte
> loop
>
> This would leave garbage in the new byte, if that matters.
>
> The other way would be to use CopyMemory, incrementing the source and
> destination pointers by different amounts. Because you have a string value
> as the first field of the UDT, you'd have to read the SAFEARRAY headers to
> get the actual starting data addresses and increments. Not a huge deal,
but
> not for the casual user either.
>
> If you don't feel like probing SAFEARRAYs, you could try using
> (VarPtr(Old|NewArray(0).Value2) - 40) for the starting values, copy
> (LenB(OldArray(0)) bytes per loop, and increment by (LenB(Old|NewArray(0))
> for each. That should work, but it needs testing.
>
> --
>
> Jim Mack
> MicroDexterity Inc
> www.microdexterity.com
>
>
>
>
>
>
| |
| Jim Mack 2005-08-30, 7:55 am |
| Tim Rude wrote:
> Jim,
>=20
> Since all my strings are fixed-length, do I need the SAFEARRAY stuff?
No, but it would avoid any conversions from Unicode to Ansi and back. =
What you're doing now probably forces those conversions on every copy. =
If this is a one-time event then it makes no difference, but for =
repeated use where speed matters, I'd avoid conversions.
As long as it's working for you, whatever you're doing is perfect :-)
--=20
Jim Mack
MicroDexterity Inc
www.microdexterity.com
| |
| Tim Rude 2005-08-30, 6:55 pm |
| This will be a one-time event that occurs when the user runs the new version
of the program for the first time. It will convert the old data file into
the new format, and will then continue to use the new format from now on
(until the next program update <g> ).
At most, it will be about 200 records (probably much less). So the speed
shouldn't be a problem. It's gotta be considerably faster than transferring
the data field by field. And easier on me to program it too.
Thanks for the input.
--
Tim Rude
timrude@NOSPAM.hotmail.com
(remove NOSPAM. for correct email address)
"Jim Mack" <jmack@mdxi.nospam.com> wrote in message
news:OX1wLsVrFHA.1028@TK2MSFTNGP12.phx.gbl...
Tim Rude wrote:
> Jim,
>
> Since all my strings are fixed-length, do I need the SAFEARRAY stuff?
No, but it would avoid any conversions from Unicode to Ansi and back. What
you're doing now probably forces those conversions on every copy. If this is
a one-time event then it makes no difference, but for repeated use where
speed matters, I'd avoid conversions.
As long as it's working for you, whatever you're doing is perfect :-)
--
Jim Mack
MicroDexterity Inc
www.microdexterity.com
| |
| Tim Rude 2005-08-30, 6:55 pm |
| I'm not sure how that would help me. It screws up the rest of the program
that uses the data since I would now have to address each element of the
data differently, i.e. if Rec is declared as UDT2:
I have to address each of the pre-existing elements as:
Rec.OldData.SomeValue
rather than:
Rec.SomeData
So I think I'll stick with the CopyMemory, which appears to work
wonderfully.
--
Tim Rude
timrude@NOSPAM.hotmail.com
(remove NOSPAM. for correct email address)
"NickHK" <TungCheWah@Invalid.com> wrote in message
news:ums0%23URrFHA.460@TK2MSFTNGP15.phx.gbl...
> Tim,
> Not sure it's any better than your CopyMem, but:
>
> Public Type UDT1
> ...
> End Type
>
> Public Type UDT2
> OldData as UDT1
> NewData as Byte 'new item added at very end of UDT
> End Type
>
> NickHK
>
>
> "Tim Rude" <timrude@nospam.hotmail.com> wrote in message
> news:#9Aa9$QrFHA.1032@TK2MSFTNGP12.phx.gbl...
etc.[color=darkred]
> to
> so
without[color=darkred]
and[color=darkred]
value[color=darkred]
to[color=darkred]
> but
(LenB(Old|NewArray(0))[color=darkred]
>
>
| |
| Larry Serflaten 2005-08-30, 6:55 pm |
|
"Tim Rude" <timrude@nospam.hotmail.com> wrote
> This will be a one-time event that occurs when the user runs the new version
> of the program for the first time. It will convert the old data file into
> the new format, and will then continue to use the new format from now on
> (until the next program update <g> ).
>
> At most, it will be about 200 records (probably much less). So the speed
> shouldn't be a problem. It's gotta be considerably faster than transferring
> the data field by field. And easier on me to program it too.
>
> Thanks for the input.
As a side note, my first impression was more like Jim's first suggestion.
Don't even bother with any types, you know you have 2030 byte records
in one file, and need to turn them into 2031 bytes records in another file.
So, write one more byte to the old file (where the last NewValue would be)
and declare a 2031 byte array. You can then step through the old file at
increments of 2030 bytes, picking up one record's worth of data and
writting it out to the new file. For 'off the cuff' code it might look like:
Dim Data(1 To 2031) As Byte
Dim DefaultNewValue As Byte
Open OldFile For Binary As OF
Open NewFile For Binary As NF
LastByte = LOF(OF)
Put OF, LastByte + 1, DefaultNewValue
For I = 1 To LastByte Step 2030
Get #OF, I, Data
Data(2031) = DefaultNewValue
Put #NF, , Data
Next
Close
No conversions, or copying data between memory is required....
LFS
| |
| Marv Wade 2005-08-30, 6:55 pm |
| Do yourself a favorite and add an additional field for further expansion
when you resize the record this time.
Marv
"Tim Rude" <timrude@nospam.hotmail.com> wrote in message
news:ecn%23HxWrFHA.3264@TK2MSFTNGP12.phx.gbl...
> This will be a one-time event that occurs when the user runs the new
> version
> of the program for the first time. It will convert the old data file into
> the new format, and will then continue to use the new format from now on
> (until the next program update <g> ).
>
> At most, it will be about 200 records (probably much less). So the speed
> shouldn't be a problem. It's gotta be considerably faster than
> transferring
> the data field by field. And easier on me to program it too.
>
> Thanks for the input.
>
> --
> Tim Rude
>
> timrude@NOSPAM.hotmail.com
> (remove NOSPAM. for correct email address)
>
> "Jim Mack" <jmack@mdxi.nospam.com> wrote in message
> news:OX1wLsVrFHA.1028@TK2MSFTNGP12.phx.gbl...
> Tim Rude wrote:
>
> No, but it would avoid any conversions from Unicode to Ansi and back. What
> you're doing now probably forces those conversions on every copy. If this
> is
> a one-time event then it makes no difference, but for repeated use where
> speed matters, I'd avoid conversions.
>
> As long as it's working for you, whatever you're doing is perfect :-)
>
> --
>
> Jim Mack
> MicroDexterity Inc
> www.microdexterity.com
>
>
>
| |
| Tim Rude 2005-08-30, 6:55 pm |
| "Marv Wade" <NG@columbus.rr.com> wrote in message
news:OrYyX1YrFHA.1128@TK2MSFTNGP11.phx.gbl...
> Do yourself a favorite and add an additional field for further expansion
> when you resize the record this time.
>
<g>
I would, except that this is a flat-file database kind of thing and I never
know what sort of additional data I might need to add to the record until I
actually need it. So I just add new items as needed.
It's worked over the last 15 yrs or so (this actually started out as a QB
program). Everytime I change the data file format, I add another chunk of
code to convert from yet another old format to the new one. The previous
change was the one where I added the arrays into the UDT. For prior
conversions I just did it field by field. But when the arrays got involved I
figured there had to be a quicker way. And there was.
--
Tim Rude
timrude@NOSPAM.hotmail.com
(remove NOSPAM. for correct email address)
| |
| Tim Rude 2005-08-30, 6:55 pm |
| Not a bad idea there. But the CopyMemory thing just seems so  . :)
--
Tim Rude
timrude@NOSPAM.hotmail.com
(remove NOSPAM. for correct email address)
"Larry Serflaten" <serflaten@usinternet.com> wrote in message
news:uWWUykYrFHA.1684@TK2MSFTNGP14.phx.gbl...
>
> "Tim Rude" <timrude@nospam.hotmail.com> wrote
version[color=darkred]
into[color=darkred]
transferring[color=darkred]
>
> As a side note, my first impression was more like Jim's first suggestion.
> Don't even bother with any types, you know you have 2030 byte records
> in one file, and need to turn them into 2031 bytes records in another
file.
>
> So, write one more byte to the old file (where the last NewValue would be)
> and declare a 2031 byte array. You can then step through the old file at
> increments of 2030 bytes, picking up one record's worth of data and
> writting it out to the new file. For 'off the cuff' code it might look
like:
>
> Dim Data(1 To 2031) As Byte
> Dim DefaultNewValue As Byte
>
> Open OldFile For Binary As OF
> Open NewFile For Binary As NF
>
> LastByte = LOF(OF)
> Put OF, LastByte + 1, DefaultNewValue
>
> For I = 1 To LastByte Step 2030
> Get #OF, I, Data
> Data(2031) = DefaultNewValue
> Put #NF, , Data
> Next
>
> Close
>
> No conversions, or copying data between memory is required....
>
> LFS
>
| |
|
| "Tim Rude" <timrude@nospam.hotmail.com> wrote in message
news:eGPNckarFHA.2592@TK2MSFTNGP09.phx.gbl...
> Not a bad idea there. But the CopyMemory thing just seems so . :)
>
just to mention, this is exactly the point "dbase" came in [many years ago]:
they introduced "named columns" inside data file [called it "dbf"], so all
data were accessed by column name, regardless of actual physical position of
column inside record, and there could be columns unused by current app, and
"re-packing" [adding/removing] columns into data file also were simple.
there are free com objects for vb fully supporting dbf files.
here is a news thread about it [harbour.dll]:
http://www.matrixlist.com/pipermail...ary/000247.html
> --
> Tim Rude
>
> timrude@NOSPAM.hotmail.com
> (remove NOSPAM. for correct email address)
>
> "Larry Serflaten" <serflaten@usinternet.com> wrote in message
> news:uWWUykYrFHA.1684@TK2MSFTNGP14.phx.gbl...
> version
> into
on[color=darkred]
speed[color=darkred]
> transferring
suggestion.[color=darkred]
> file.
be)[color=darkred]
at[color=darkred]
> like:
>
>
| |
| J French 2005-08-31, 7:55 am |
| On Tue, 30 Aug 2005 09:33:35 -0500, "Tim Rude"
<timrude@nospam.hotmail.com> wrote:
>I'm not sure how that would help me. It screws up the rest of the program
>that uses the data since I would now have to address each element of the
>data differently, i.e. if Rec is declared as UDT2:
>
>I have to address each of the pre-existing elements as:
>
> Rec.OldData.SomeValue
>
>rather than:
>
> Rec.SomeData
>
>So I think I'll stick with the CopyMemory, which appears to work
>wonderfully.
Nick's suggestion was the best one
- by far
- if it is restricted to the once a year file conversion routine then
it should have absolutely no effect on the rest of your App
It just means that you have two extra UDT declarations
- the 'new' current UDT used by all your App
- the old UDT
- the utility UDT
If only the annual update routine uses the last two UDTs then nothing
is affected
Another method is to simply update the file by knowing the old and new
record lengths, but you'll need to know what fill characters to use
- typically Chr$(0) or Chr$(32)
I would definitely use Nick's method in preference to CopyMemory
- it is self documenting and not at all arcane
| |
| J French 2005-08-31, 7:55 am |
| On Tue, 30 Aug 2005 16:21:52 -0500, "Tim Rude"
<timrude@nospam.hotmail.com> wrote:
>"Marv Wade" <NG@columbus.rr.com> wrote in message
>news:OrYyX1YrFHA.1128@TK2MSFTNGP11.phx.gbl...
[color=darkred]
><g>
>I would, except that this is a flat-file database kind of thing and I never
>know what sort of additional data I might need to add to the record until I
>actually need it. So I just add new items as needed.
>It's worked over the last 15 yrs or so (this actually started out as a QB
>program). Everytime I change the data file format, I add another chunk of
>code to convert from yet another old format to the new one. The previous
>change was the one where I added the arrays into the UDT. For prior
>conversions I just did it field by field. But when the arrays got involved I
>figured there had to be a quicker way. And there was.
Personally I would make a slightly more complex structure for the file
- something a bit similar to an INI file
It would run fractionally slower ( undetectably slower for reading say
10 records ) but it will make things totally flexible
You can add a field without doing /anything/ to the data file
|
|
|
|
|