Repairing Damaged Memo Fields

Even if the data in your .DBF file is undamaged, you won't be
able to open your .DBF file if its .DBT file is missing.  We can
help.

-----

The text you enter into memo fields is stored, separately from
the rest of the fields, in a memo text file with a .DBT
extension. The memo file has a very simple structure.

Memo files

The entire file is divided into blocks of 512 characters. Each
block is referenced by a sequential number, beginning at zero.
Block zero begins with a four-byte number in
least-significant-byte-first hexadecimal format. This number
specifies the number of the next available block. This is, in
effect, a pointer to the end of the memo file. The remainder of
block zero is not used.

Memo text is stored in as many contiguous blocks as required to
contain the text. The end of the text in each memo is marked with
two end-of-file characters (1Ah). Any bytes following the
end-of-file characters in the last block of each memo are
"undefined."  In practice, these bytes tend to hold
garbagememory images from previous dBASE operations.

There is a one-way link between database (.DBF) files and memo
(.DBT) files. The .DBF file record structure allocates ten bytes
to each memo field to hold the number of the block in which the
memo text begins. This is a one-way link because the database
record points to the memo block, but there is no information in
the memo file that points back to the database record.

What can go wrong

Memo files are not as vulnerable to many of the errors which can
damage a .DBF file. Embedded end-of-file and null characters will
damage only the memo in which they appear and can be easily
edited out of the memo. Disk errors, such as lost clusters and
bad sectors, can usually be corrected by running CHKDSK and then
editing the affected memos.

The most prevalent kind of data damage involving memo files is
caused by operator error. The .DBF and .DBT files are a set, and
should be treated as such. When files are backed up, restored,
renamed, or copied, it is important to perform identical
procedures on both files.

For example, if you restore the .DBF file from a backup but not
the .DBT file, you are left in one of two situations. If dBASE
finds a previous copy of the .DBT file, the database and memos
may be dissociatedrecords point to incorrect or nonexistent
memos. The other possibility is that there is no .DBT file at
all, in which case dBASE cannot even USE the file.

Recovering dissociated memo files is a time-consuming, laborious
process. You should attempt it only if you need the data in the
memo file and there is no other way to reproduce it.

Replacing lost memo files

If the memo file is entirely missing, recovery requires that you
provide a replacement .DBT file so that dBASE can USE the
database. The method of recovery is simple, but the loss may be
devastating.

When you attempt to open a database file with memo fields and the
corresponding memo file is not available, dBASE returns the error
message, ".DBT file cannot be opened."  

When this occurs, the first thing you should do is look for the
.DBT file. Many times the file exists, but dBASE doesn't see it
for some reason. Did you forget to restore it from the backup? 
Is it on the disk under a different name, or is it in a different
directory on the disk?  Note that the ".DBT" extension is the
only default file extension that cannot be substituted in dBASE.

If there is no .DBT file, look for a file with the same name and
a .TBK extension. This file is a backup of the .DBT file, created
when you changed the database structure with MODIFY STRUCTURE or
CREATE/MODIFY SCREEN. If this file exists, rename it with the
.DBT extension. This will permit access to the database file, and
the memo fields that were present when the .TBK file was created.
Any memo field changes or additions since the backup was made
will be lost. Depending on the degree and type of changes made to
the database file, the memos from a backup of a .DBT file might
be dissociated from the database. If this is the case, you must
decide whether to start over with the memo file or proceed to the
next section on recovering text from memo files.

Finally, if no memo file exists, you've no choice but to accept
your losses and go on. However, dBASE needs a .DBT file in order
to open the .DBF file. This is a minor problem that can be solved
with the following commands at the DOS prompt:

COPY CON: <yourfile>.DBT
DUMMY <RETURN>
<CTRL-Z> <RETURN>

This sequence of commands creates a .DBT file that dBASE can open
along with the .DBF file. 

Once you're able to USE the file, MODIFY STRUCTURE to remove the
memo fields. This step is important, since the current memo
pointers in the database records point to nonexistent memos. If
desired, use MODIFY STRUCTURE a second time to redefine the memo
fields.

Recovering text from memo files

There are occasions when you have a memo file that contains good
memo text that you are unable to access because the database and
memo files have become dissociated. A similar situation exists
when the .DBF file has been lost or erased. If the contents of
the memo file are valuable you might want to attempt recovering
the memos. 

Be forewarned, this process involves a great deal of work. You
will need Xlate (see page xx), a text editor, and a great deal of
patience. With these tools you can create a single file
containing the text of all memos that were in the .DBT file. If
you want to go the extra step to reconstruct the .DBT file, you
will also need the database (.DBF) file and an extra measure of
patience.

Xlate can produce an editable file from the dissociated memo file
if you set its options like those in Table 1.

In Xlate's translation table, change the -1 assigned to ASCII 138
to 10 and the -1 at ASCII 141 to 13. This will cause Xlate to
translate dBASE's soft carriage return/linefeed characters to
values understood by most editors.

It's a good idea to alter Xlate's translation table to translate
end-of-file markers (ASCII 26) to some unique character you can
search for with your text editor. One character that's suited to
this purpose is the pipe character ("|", ASCII 124), which is not
commonly found in text. This will make it quite simple to find
the end of a memo in the large text file.

When the Xlate translation has done its job, load the translated
file into your text editor and take a look. You can expect to
find a block of random characters at the beginning of the file,
and a similar block between memos. 

Because of the way dBASE handles memo file edits, you may also
encounter several versions of memos within the text file. The
more recent copies of a memo always appear later in the the text
file. For this reason, you may want to begin your editing at the
end of the file and work your way back to the beginning, deleting
garbage characters and out-dated versions of memos as you go. 

It sounds like a lot of work, and it is. When you're through with
this initial edit, you should have a clean text file containing
all the memos that were recovered.

Reconstructing the .DBT file is a formidable task. dBASE III PLUS
provides no mechanism to put text into a memo field without
actually editing the memo field. However, once you're in the memo
field, you can read in text from another file.

If you use the dBASE text editor for memo fields, you will need
to prepare by breaking the single edited file into separate
filesone for each memo. Depending on the number of memos in the
text file, you may want (or need) to approach this process in
sections to limit the number of files you work with
simultaneously. When you have created the individual files, edit
each memo field in the database file and read in the
corresponding text file with the editor's Ctrl-KR command. 

If you've specified your own text editor for memo fields in
CONFIG.DB, you may be able to read portions of the large file
directly into the memo field. dBASE expects external text editors
to edit a file named Dbasedit.TMP. Once dBASE has passed control
to your text editor, you can use whatever method is easiest for
you to save the correct text in Dbasedit.TMP. This generally
means you won't have to break up the large text file. 

The entire file is handled in blocks of 512 bytes. Each block is
numbered sequentially, 1, 2, 3, and so forth.

The first block is the header and can be considered block number
zero. dBASE III PLUS uses only the first four bytes of the
header. They contain the number of the next available 512-byte
block. This number is in least-significant-byte-first format (See
Table).

Each memo field of each record in the .DBF file contains the
number of the block (in ASCII) where the memo field data is
located. If the memo field contains no data, there is no number
in the .DBF file. When you save a memo, the next available block
is used, and its number is stored in ASCII in the memo field of
that record, invisibly behind the word memo.

When you change a memo, the old contents remain in the .DBT file,
the entire changed memo is added to the next available block, and
the number in the .DBF is changed to reflect the new location.
This leaves the old block unused, without a pointer to it in the
.DBF, and explains why .DBT files grow to such large sizes when
edited repeatedly. When the file is copied from within dBASE with
USE <Filename> and COPY TO <Filename>, only the blocks specified
by the current memo field pointers are copied, thus discarding
blocks no longer used and recovering the disk space.

If the memo uses more than one 512-byte block, the next available
block number is incremented by the number of blocks used. In
other words, if the next available block is number 15, after a
1200 byte memo is added, the next available block is 18. Each
memo terminates with two 1Ah characters. If a memo contains 520
bytes, the ninth and tenth bytes of the next block will hold
these terminators. The remaining 502 bytes of the block will be
unused.
