| Author |
Converting Clarion DAT files to TPS
|
|
| Jason Johnson 2005-01-10, 3:55 pm |
| I have re-written my DOS app in C5b and have just written a data conversion
tool to convert from the old DAT files to TPS files. The files is large
(13,000 records) and each record has about 200 fields (large in my opinion).
The conversion start and is converting about 50 records per second, which
seems fine and quick to me. Then, at about half-way...BAM... it slows to
about 5 records per second. It doesn't gradually slow down, it just acts as
if it hit a wall. Any ideas why it would start quick and then just suddenly
slow down like that.
My machine is 800MHz and its Win98SE. I'm not too worried about it since it
gets the job done, I just was wondering if anyone might know why it would do
that. thanks for any suggestions.
--
Jason
jasonfive@att.net
| |
| Jason Johnson 2005-01-10, 3:55 pm |
| Thanks for the reply. Another question. Is it faster to do the APPEND with
the BUILD at the end rather than just doing the ADD where it updates as it
goes?
"JMumf" <jmmfrd@yahoo.com> wrote in message
news:1105054310.323028.8920@f14g2000cwb.googlegroups.com...
> Jason:
>
> My guess is it has something to do with the Indexes. I hope you don't
> have a huge number of indexes.
> If you do see if you can remove some of them
>
> To make this fast:
> Use Append instead of ADD. Then about every 1000 records do a BUILD.
>
> Append(MyTPSTable)
> AppendCnt+=1
> IF AppendCnt%1000=0
> BUILD(MyTPSTable)
> END
>
> !At the END
> BUILD(MyTPSTable) !Make sure this is done at the end
>
> Another way to make it fast Logout, Commit..
>
> After Open and Begining.
> LOGOUT(1,MyTPSTable)
>
> ADD(MyTPSTable)
> AddCnt+=1
> IF AddCnt%1000=0
> COMMIT
> LOGOUT(1,MyTPSTable)
> END
>
> !At the END
> COMMIT !Make sure this is done at the end
>
> Either will be much faster. LOGOUT might be the easiest.
> I would remove STREAM and FLUSH on the OUTPUT file.
> Jim Mumford
>
> Make sure you PrimaryKey is unique.
>
> Jim Mumford
>
| |
|
| Jason:
On a TopSpeed Table I would not do an APPEND and Build for the entire
file. The Topspeed tables have a feature to only BUILD the ones that
have been Appended since the last BUILD. This way you don't sit for a
long time while it builds the whole file. I would do a BUILD every
1000 records or so.
The Logout and Commit also works well for what you are doing. It is
very fast. Again I would do a comit ever 1000 reccords or so the a new
Logout. If you do this you should continue to use the ADD. It will be
very fast.
Jim
| |
| Jason Johnson 2005-01-11, 3:55 pm |
| Thanks for the help. I used the LOGOUT/COMMIT and it works great. I tried
the APPEND/BUILD for comparison but it was much slower. I also took out the
COMMIT every 1000 records, but when I did the COMMIT only at the end, it
took quite a while for the disk to write, so I put back in the 1000 rec
COMMIT.
Great advice, thanks for the help.
"JMumf" <jmmfrd@yahoo.com> wrote in message
news:1105385579.406387.68850@c13g2000cwb.googlegroups.com...
> Jason:
>
> On a TopSpeed Table I would not do an APPEND and Build for the entire
> file. The Topspeed tables have a feature to only BUILD the ones that
> have been Appended since the last BUILD. This way you don't sit for a
> long time while it builds the whole file. I would do a BUILD every
> 1000 records or so.
>
> The Logout and Commit also works well for what you are doing. It is
> very fast. Again I would do a comit ever 1000 reccords or so the a new
> Logout. If you do this you should continue to use the ADD. It will be
> very fast.
>
> Jim
>
| |
|
| Glad I could help.
My guess is that the LOGOUT/COMMIT is at least 10 times faster than
ADD with STREAM and FLUSH.
Thanks for letting everyone know what worked.
Jim Mumford
| |
|
| this newsgroup on google is basicly not updated and been seen by the
normal newsgroup readers like outlook, there is where the clarion
activity is going on, not on this google
for login:
news.softvelocity.com
:-)
menno
| |
| Jason Johnson 2005-01-12, 8:55 pm |
| Thanks for the reply. Another question. Is it faster to do the APPEND with
the BUILD at the end rather than just doing the ADD where it updates as it
goes?
"JMumf" <jmmfrd@yahoo.com> wrote in message
news:1105054310.323028.8920@f14g2000cwb.googlegroups.com...
> Jason:
>
> My guess is it has something to do with the Indexes. I hope you don't
> have a huge number of indexes.
> If you do see if you can remove some of them
>
> To make this fast:
> Use Append instead of ADD. Then about every 1000 records do a BUILD.
>
> Append(MyTPSTable)
> AppendCnt+=1
> IF AppendCnt%1000=0
> BUILD(MyTPSTable)
> END
>
> !At the END
> BUILD(MyTPSTable) !Make sure this is done at the end
>
> Another way to make it fast Logout, Commit..
>
> After Open and Begining.
> LOGOUT(1,MyTPSTable)
>
> ADD(MyTPSTable)
> AddCnt+=1
> IF AddCnt%1000=0
> COMMIT
> LOGOUT(1,MyTPSTable)
> END
>
> !At the END
> COMMIT !Make sure this is done at the end
>
> Either will be much faster. LOGOUT might be the easiest.
> I would remove STREAM and FLUSH on the OUTPUT file.
> Jim Mumford
>
> Make sure you PrimaryKey is unique.
>
> Jim Mumford
>
| |
| Jason Johnson 2005-01-13, 3:55 pm |
| Thanks for the help. I used the LOGOUT/COMMIT and it works great. I tried
the APPEND/BUILD for comparison but it was much slower. I also took out the
COMMIT every 1000 records, but when I did the COMMIT only at the end, it
took quite a while for the disk to write, so I put back in the 1000 rec
COMMIT.
Great advice, thanks for the help.
"JMumf" <jmmfrd@yahoo.com> wrote in message
news:1105385579.406387.68850@c13g2000cwb.googlegroups.com...
> Jason:
>
> On a TopSpeed Table I would not do an APPEND and Build for the entire
> file. The Topspeed tables have a feature to only BUILD the ones that
> have been Appended since the last BUILD. This way you don't sit for a
> long time while it builds the whole file. I would do a BUILD every
> 1000 records or so.
>
> The Logout and Commit also works well for what you are doing. It is
> very fast. Again I would do a comit ever 1000 reccords or so the a new
> Logout. If you do this you should continue to use the ADD. It will be
> very fast.
>
> Jim
>
| |
| Jason Johnson 2005-01-15, 8:55 am |
| Thanks for the reply. Another question. Is it faster to do the APPEND with
the BUILD at the end rather than just doing the ADD where it updates as it
goes?
"JMumf" <jmmfrd@yahoo.com> wrote in message
news:1105054310.323028.8920@f14g2000cwb.googlegroups.com...
> Jason:
>
> My guess is it has something to do with the Indexes. I hope you don't
> have a huge number of indexes.
> If you do see if you can remove some of them
>
> To make this fast:
> Use Append instead of ADD. Then about every 1000 records do a BUILD.
>
> Append(MyTPSTable)
> AppendCnt+=1
> IF AppendCnt%1000=0
> BUILD(MyTPSTable)
> END
>
> !At the END
> BUILD(MyTPSTable) !Make sure this is done at the end
>
> Another way to make it fast Logout, Commit..
>
> After Open and Begining.
> LOGOUT(1,MyTPSTable)
>
> ADD(MyTPSTable)
> AddCnt+=1
> IF AddCnt%1000=0
> COMMIT
> LOGOUT(1,MyTPSTable)
> END
>
> !At the END
> COMMIT !Make sure this is done at the end
>
> Either will be much faster. LOGOUT might be the easiest.
> I would remove STREAM and FLUSH on the OUTPUT file.
> Jim Mumford
>
> Make sure you PrimaryKey is unique.
>
> Jim Mumford
>
| |
|
| Glad I could help.
My guess is that the LOGOUT/COMMIT is at least 10 times faster than
ADD with STREAM and FLUSH.
Thanks for letting everyone know what worked.
Jim Mumford
| |
|
| Glad I could help.
My guess is that the LOGOUT/COMMIT is at least 10 times faster than
ADD with STREAM and FLUSH.
Thanks for letting everyone know what worked.
Jim Mumford
| |
|
| this newsgroup on google is basicly not updated and been seen by the
normal newsgroup readers like outlook, there is where the clarion
activity is going on, not on this google
for login:
news.softvelocity.com
:-)
menno
|
|
|
|