Clipper - Memo field

Author

Memo field

Robert J. Stuart

2006-02-08, 6:55 pm

Has anyone experienced a limit to the size of a .DBT file. It seems that in
a rather large database, where the .DBT file has now reached a size of 15MB,
It will not accept any more memo entries unless I delete some of the old
ones. Sort of like it has reached a maximum limit.

Bob S.

s_boomer

2006-02-08, 6:55 pm

Rob
Maybe it is time to consider using FlexfileII for your memo storage. I
currently have applicaions running with multiple Flexfiles of over 20M
in text and over 1.5G of images
Sam

Robert J. Stuart

2006-02-08, 6:55 pm

Thanks guys. That's what I thought. The .DBT file is at 16MB and will not
let me save any more memos. This particular application is an old S87. I
figure what I will do is recompile it using 5.3. But I did learn something
interesting. In my course of trying to figure out how to recover some space
in the DBT file, I did a "copy to" as a backup of the original files and I
noticed that the new .DBT was now only 4 MB in size.

I thought maybe the entire file didn't get copied, so I tried it again.
Again the file size on the .DBT reduced in size by almost 12MB. Now I am
not sure what the "copy to" routine did. Perhaps there was a lot of wasted
space there and it filtered it out in the copy process. But that sort of
solved my immediate problem.

Anyone have any ideas on what is going on? By the way when I say "copy to",
I mean the clipper command copy to, not the windows or dos command. Doing a
windows or dos copy just resulted in an exact duplicate in file size.

Bob S.

"Robert J. Stuart" <rjstuart@earthlink.net> wrote in message
news:laqGf.12481$rH5.1370@newsread2.news.atl.earthlink.net...
> Has anyone experienced a limit to the size of a .DBT file. It seems that
in
> a rather large database, where the .DBT file has now reached a size of
15MB,
> It will not accept any more memo entries unless I delete some of the old
> ones. Sort of like it has reached a maximum limit.
>
> Bob S.
>
>

Stephen Quinn

2006-02-08, 6:55 pm

Robert

Max. size for a DBT is 32M, although I believe in early versions of Clipper it
was 16M.

HTH
Steve

Ross McKenzie

2006-02-08, 6:55 pm

On Wed, 08 Feb 2006 17:49:05 GMT, "Robert J. Stuart"
<rjstuart@earthlink.net> wrote:

>Has anyone experienced a limit to the size of a .DBT file. It seems that in
>a rather large database, where the .DBT file has now reached a size of 15MB,
>It will not accept any more memo entries unless I delete some of the old
>ones. Sort of like it has reached a maximum limit.
>
>Bob S.
>
>

Hello Bob.

S'87 had a 16 Mbyte limit before the 2am version was released and
lifted the limit to 32Mbytes.

Regards,

Ross McKenzie
ValuSoft
Melbourne Australia

valusoft AT optusnet DOT com DOT au

ron

2006-02-08, 6:55 pm

Hi Robert,
Using the clipper copy to command is indeed a good way to shrink memo
files.
As I remember, memo files use 4kb pages, and each change in memo text
record adds a new version of the original record. Using copy to only
copies the most recent version and leaves out the exess empty space.
[color=darkred]
> In my course of trying to figure out how to recover some space
> in the DBT file, I did a "copy to" as a backup of the original files and I
> noticed that the new .DBT was now only 4 MB in size.
>
> I thought maybe the entire file didn't get copied, so I tried it again.
> Again the file size on the .DBT reduced in size by almost 12MB. Now I am
> not sure what the "copy to" routine did. Perhaps there was a lot of wasted
> space there and it filtered it out in the copy process. But that sort of
> solved my immediate problem.
>

R. Totale

2006-02-08, 6:55 pm

Nick Ramsay

2006-02-09, 3:55 am

On 8 Feb 2006 14:40:09 -0800, "ron" <ronbroere@bbkm.nl> wrote:

>Hi Robert,
>Using the clipper copy to command is indeed a good way to shrink memo
>files.
>As I remember, memo files use 4kb pages, and each change in memo text
>record adds a new version of the original record. Using copy to only
>copies the most recent version and leaves out the exess empty space.
>

Nearly correct. They're 512 byte records actually. Once written, the
..DBT record(s) are never updated, merely dereferenced in the .dbf memo
pointer field. The updated text is added to the end of file and the
memo pointer is updated to point to the start of the new memo
record(s). This leads to enormous bloat if the memo date is edited
frequently. Using COPY TO is the only way to re-write the .DBT
without the dereferenced records.

It's a horrible kludge, but Wayne Ratliff only added the .DBT support
in the w

before dBase III shipped, so I guess he didn't have time
to make it sophisticated ;)

Incidentally, the reason that the early implementation was 16Mb is
that Wayne used a signed integer for the record pointer.

AnthonyL

2006-02-09, 6:55 pm

On Wed, 08 Feb 2006 22:32:35 GMT, "Robert J. Stuart"
<rjstuart@earthlink.net> wrote:

>Thanks guys. That's what I thought. The .DBT file is at 16MB and will not
>let me save any more memos. This particular application is an old S87. I
>figure what I will do is recompile it using 5.3. But I did learn something
>interesting. In my course of trying to figure out how to recover some space
>in the DBT file, I did a "copy to" as a backup of the original files and I
>noticed that the new .DBT was now only 4 MB in size.
>
>I thought maybe the entire file didn't get copied, so I tried it again.
>Again the file size on the .DBT reduced in size by almost 12MB. Now I am
>not sure what the "copy to" routine did. Perhaps there was a lot of wasted
>space there and it filtered it out in the copy process. But that sort of
>solved my immediate problem.
>
>Anyone have any ideas on what is going on? By the way when I say "copy to",
>I mean the clipper command copy to, not the windows or dos command. Doing a
>windows or dos copy just resulted in an exact duplicate in file size.
>

You are spot on. It was bloat because delete didn't really
delete/pack in the .dbt.

Our S'87 has a pack/reindex routine that actual does a "copy for not
deleted" then renames back (at DOS level) to the original for any data
files with associated .dbt.

If you hadn't done it before and have a lot of record movement your
results are pretty consistent.

However just try a dump of the .dbt to a text file and compare the
results to give yourself peace of mind or write a little routine to
compare the contents of original .dbt with new .dbt for non-deleted
records.

Good luck

--
AnthonyL

Robert J. Stuart

2006-02-09, 6:55 pm

Nick Thanks for the info. It seems as if there is a lot of junk is being
written in-between each record. I looked at the .DBT file with notepad and
found all this junk in-between each record. Here is a short cut and paste.
I have Xed out private substance.

®w T. I TRIED HIS OFFICE (xxx-xxxx, EXT xxx) AND
HOME (xxx-xxxx). NO ANSWER. LEFT MESSAGE BOTH PLACES TO CALL US BACK. SS
(R) CLIPSORT CLIPSORT.TMP disk full <<NMSG>> R6002
- floating point not loaded
Rr
- null pointer ass
äcý : MATH
- floating-point error: e invalid
g divide by 0
h overflow
n stack overflow
p explicitl To-±iüX.TrüXÀ,
q= èâFrÞÒü äcèèâFrÞæü Sk @kdÔ¥
ÿüXSkìü dn!;?röRi<d d Ì i<d I
oD Sk 84/20/98 VOICE MAIL LJA (R) CLIPSORT CLIPSORT.TMP
disk full <<NMSG>> R6002
- floating point not loaded
R6001
- null pointer assignment
ý : MATH
- floating-point error: e invalid
g divide by 0
h overflow
n stack overflow
p explicitly generated
R6000
- stack overflow
R6003
- integer divide by 0
R6009
- not enough space for environment
ü
ÿ run-time error x DOMAIN error
y SING error
z TLOS12/18/98 PER xxxx xxxx - IS REFUSING TRTMENT. ALSO WE AERE TRYING TO
CLARIFY
#OF DWI'S 2 OR 3. NO DIVING RECORD IN FILE, I WILL FAX COPY OF DS449 PART 1
AS SHE DIDN'T KEEP COPY LJA (R) CLIPSORT CLIPSORT.TMP disk
full <<NMSG>> R6002
- floating point not loaded
R6001
- null pointer assignment
ý : MATH
- floating-point error: e invalid
g divide by 0
h overflow
n stack overflow
p explicitly generated
R6000
- stack overflow
R6003
- integer divide by 0
R6009
- n
(R) CLIPSORT CLIPSORT.TMP disk full <<NMSG>> R6002
- floating point not loaded

What is with the:

- null pointer assignment
ý : MATH
- floating-point error: e invalid
g divide by 0
h overflow
n stack overflow
p explicitly generated
R6000
- stack overflow
R6003
- integer divide by 0
R6009

in between each record?

Bob S.

"Nick Ramsay" <news@nospam.demon.co.uk> wrote in message
news:hd2mu15c95sa6deq70o5holunsto0tpq4t@
4ax.com...
> On 8 Feb 2006 14:40:09 -0800, "ron" <ronbroere@bbkm.nl> wrote:
>
>
> Nearly correct. They're 512 byte records actually. Once written, the
> .DBT record(s) are never updated, merely dereferenced in the .dbf memo
> pointer field. The updated text is added to the end of file and the
> memo pointer is updated to point to the start of the new memo
> record(s). This leads to enormous bloat if the memo date is edited
> frequently. Using COPY TO is the only way to re-write the .DBT
> without the dereferenced records.
>
> It's a horrible kludge, but Wayne Ratliff only added the .DBT support
> in the w

before dBase III shipped, so I guess he didn't have time
> to make it sophisticated ;)
>
> Incidentally, the reason that the early implementation was 16Mb is
> that Wayne used a signed integer for the record pointer.

Nick Ramsay

2006-02-09, 6:55 pm

On Thu, 09 Feb 2006 15:22:15 GMT, "Robert J. Stuart"
<rjstuart@earthlink.net> wrote:

>Nick Thanks for the info. It seems as if there is a lot of junk is being
>written in-between each record. I looked at the .DBT file with notepad and
>found all this junk in-between each record. Here is a short cut and paste.

It's because memory isn't being cleared of its contents after it's
been allocated (the memset() function in C) by the .dbf driver code.
This results in the garbage you're seeing. It's nothing to worry
about (although it *is* sloppy programming IMHO).

Ray Marron

2006-02-09, 6:55 pm

Robert J. Stuart wrote:
> Nick Thanks for the info. It seems as if there is a lot of junk is being
> written in-between each record. I looked at the .DBT file with notepad and
> found all this junk in-between each record. Here is a short cut and paste.
[SNIP]

That is most likely either what was already on disk at those locations
when the file size was grown (the memo knows how long it really is from
the first few bytes of each one) or whatever was in the memory that was
allocated (but not initialized) for the buffer to write the new memo
blocks to the file. Really makes you wonder what kind of sensitive
information is sitting in uninitialized chunks of all kinds of files,
doesn't it?

--
Ray Marron

pete@nospam.demon.co.uk

2006-02-10, 3:55 am

In article <4asmu15i0c44df5mf3upasddq2br70l6o7@4ax.com>
delete-all-spam3@witzendcs.co.uk "Nick Ramsay" writes:

> On Thu, 09 Feb 2006 15:22:15 GMT, "Robert J. Stuart"
> <rjstuart@earthlink.net> wrote:
>
>
> It's because memory isn't being cleared of its contents after it's
> been allocated (the memset() function in C) by the .dbf driver code.
> This results in the garbage you're seeing. It's nothing to worry
> about (although it *is* sloppy programming IMHO).

Sloppy indeed; one only has to DEBUG MS Word docs to see that
such sloppiness continues twenty years on... All manner of
sensitive information could be sitting there in plain(ish) view
to those that look.

I hardly ever use memo fields, but if one's applications are
supplied with preloaded .dbt files it's a good idea to clean them
up with COPY TO before distribution.

Pete
--
"We have not inherited the earth from our ancestors,
we have borrowed it from our descendants."

Nick Ramsay

2006-02-10, 3:55 am

On Fri, 10 Feb 2006 06:19:35 +0000 (UTC), pete@nospam.demon.co.uk
wrote:

>In article <4asmu15i0c44df5mf3upasddq2br70l6o7@4ax.com>
> delete-all-spam3@witzendcs.co.uk "Nick Ramsay" writes:
>
>
>Sloppy indeed; one only has to DEBUG MS Word docs to see that
>such sloppiness continues twenty years on... All manner of
>sensitive information could be sitting there in plain(ish) view
>to those that look.
>

Indeed - some of the docs that come out of M$ themselves can be
instructive in this regard ;)

>I hardly ever use memo fields, but if one's applications are
>supplied with preloaded .dbt files it's a good idea to clean them
>up with COPY TO before distribution.
>

Hi Pete!

Unfortunately, even doing a COPY TO isn't going to solve this problem.
Once the .dbt record is written (uninitialised garbage-collected
memory & all), it will stay in the .dbt for anyone with a hex editor
to go looking at.

<grumpy_old_man>

I just cannot understand why programmers have such an aversion to
memset()ing malloc()'d memory before use if it's subsequently going to
be used to write data back to disk. This same aversion caused all
sorts of instability in the beta-test versions of Clipper 5 in (gulp,
was it really 16 years ago?) 1990. I produced a workaround (and a
rant!) at the time - still available here for those interested:

http://www.witzendcs.co.uk/html/files/ramclear.zip

</grumpy_old_man>

pete@nospam.demon.co.uk

2006-02-10, 6:55 pm

In article <eemou11n6k70cjmssmg3b3ri19fqub8492@4ax.com>
delete-all-spam3@witzendcs.co.uk "Nick Ramsay" writes:

> On Fri, 10 Feb 2006 06:19:35 +0000 (UTC), pete@nospam.demon.co.uk
> wrote:
[..]
>
> Hi Pete!

Hi Nick!
>
> Unfortunately, even doing a COPY TO isn't going to solve this problem.
> Once the .dbt record is written (uninitialised garbage-collected
> memory & all), it will stay in the .dbt for anyone with a hex editor
> to go looking at.

Ah -- I didn't know it was *that* bad. I guess the workaround
would be to write memos in 512-byte (or whatever) blocks...

> <grumpy_old_man>
>
> I just cannot understand why programmers have such an aversion to
> memset()ing malloc()'d memory before use if it's subsequently going to
> be used to write data back to disk.

Or even using calloc() and accepting the overhead of an extra
parameter? I guess it's the same sloppy frame of mind that that
sometimes forgets to free() those same allocations. Presumably
this has been sorted in [x]Harbour.

> This same aversion caused all
> sorts of instability in the beta-test versions of Clipper 5 in (gulp,
> was it really 16 years ago?) 1990. I produced a workaround (and a
> rant!) at the time - still available here for those interested:
>
> http://www.witzendcs.co.uk/html/files/ramclear.zip

And useful for not only Clipper!
>
> </grumpy_old_man>

Grumpiness comes with maturity; I'm looking forward to the "total
intolerance" stage next ;-)

Pete
--
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

Sponsored Links