This file explains some of the low-level details of the overlay manager.
It is assumed that the reader is familiar with the documentation provided
in the Turbo Pascal manuals relating to overlays, and with the use of
overlays in general.


Overview

Each overlaid unit has three parts: a static "dispatcher" segment, the
actual overlaid code, and a transient fixup table.  The static dispatcher
for each unit is a small code segment that always remains in memory.  It
transfers control to the overlay manager if the unit is not currently in
memory, or directly to the unit in the overlay buffer if it is already
present.  The overlaid code itself is stored in the application's OVR
file, which is read by the overlay manager each time it must reload an
overlay.  The fixup table is also stored in the OVR file, and is used to
correct intersegment references in the overlay code, based on the starting
address of the program.


The Static Dispatcher

Each static dispatcher is a small code segment that's linked into the EXE
file.  A unique dispatcher always remains in memory for each overlaid
unit.  Whenever you call a procedure in an overlaid unit, you're really
calling a tiny (5 byte) routine in the static dispatcher.

If you have the Turbo Pascal run-time library source, you can find the
"official" documentation for this structure in SE.ASM, which is in INCLUDE.ARC.

Each dispatcher begins with a 32 byte header area, which is laid out as
follows:

type
  OverlayHeader = record
    {00} ReturnInt : Word;
    {02} ReturnOfs : Word;
    {04} FileOfs : LongInt;
    {08} CodeSize : Word;
    {0A} FixupSize : Word;
    {0C} EntryPts : Word;
    {0E} CodeListNext : Word;
    {10} LoadSegment : Word;
    {12} Reprieved : Word;
    {14} LoadListNext : Word;
    {16} EmsPage : Word;
    {18} EmsOffset : Word;
    {1A} UserData : array [0..2] of Word;
  end;

ReturnInt and ReturnOfs are used when RETurning to a unit that has been
booted out by another unit.  ReturnInt always contains an INT $3F
instruction.  ReturnOfs is a variable field.  These are explained in
more detail later.

FileOfs is the offset of the overlaid code segment within the overlay file.
The lowest value this can have is 4 because each overlay file starts with a
four byte signature.

CodeSize specifies the number of bytes of actual code for this unit.  To
determine the number of bytes required in the overlay buffer to load the
code of the unit this must be rounded up to the next paragraph.

FixupSize denotes the transient space required while loading the overlay
unit.  This must also be rounded up to the next paragraph.

EntryPts specifies the number of 5-byte jump vector entries in the static
dispatcher.  Although you might think that this would be the same as the number
of interfaced procedures and functions in the unit, that's not true.  The
static dispatcher has an entry for _every_ procedure and function in the
unit, whether it's global or local, interfaced or hidden.  The reason
becomes clear when you think about passing procedure parameters -- even a
non-interfaced routine can be called indirectly from outside of the unit,
and the overlay manager must still be informed when to load the code in
such a case.  This number thus provides a count of all the routines in a
unit.  (It also adds one for the initialization block, if any.)

CodeListNext contains the static segment address of the next unit in a linked
list of all overlaid units in the program.  The System unit variable
OvrCodeList contains the segment of the first unit in the list.  A value of
zero in CodeListNext marks the end of the list.  Segment addresses are
relative to PrefixSeg+$10.

LoadSegment is initially zero.  When the overlay manager loads the unit
from disk it stores the segment address where the unit was loaded here.
If the overlay manager unloads the unit from memory, it sets LoadSegment
back to zero.

Reprieved is a flag that is set to 1 when a unit that was on probation is
reprieved.

LoadListNext contains the static segment address of the next unit in a linked
list of all units that are currently loaded in the overlay buffer.  The
System unit variable OvrLoadList contains the segment of the first unit in
the list.  A value of zero in LoadListNext marks the end of the list.
Segment addresses are absolute.

EmsPage contains the EMS physical-page number where the unit begins when the
overlay file is stored in EMS memory.

EmsOffset contains the byte offset from the beginning of the EMS page when the
overlay file is stored in EMS memory.

UserData is not used by the overlay system as far as anyone has been able to
determine.


The header is followed by a 5-byte "vector" for each procedure and function in
the unit.  If the overlay isn't currently loaded, or is on probation, the
vector consist of an INT $3F instruction, followed by one word of data that
specifies the code offset of the routine being called, and an unused
zero-byte.  This is the state of the dispatcher when the program is initially
loaded.  The INT $3F serves to transfer control to the overlay manager's
code.

If the overlay is already loaded into memory, and is not on probation, the
vector is just a FAR JMP instruction that immediately transfers control to
the real code in the overlay buffer.  Returning from an overlaid unit is done
with a simple RETF instruction, just like with normal, non-overlaid routines,
as the FAR JMP does not affect the return address on stack pushed by the
original CALL.


The Overlay File

The overlaid code itself is stored in the application's OVR file, which is
read by the overlay manager each time it must reload an overlay.  If EMS
overlays are activated, the OVR file is read entirely into EMS during
initialization, and thereafter overlays are transferred to the overlay
buffer from EMS rather than from disk.

The transient fixup tables are also stored in the OVR file, immediately
following the code for each unit.  The overlay manager reads this
information into the overlay buffer each time an overlay is loaded.  Fixup
information is used to correct intersegment references in the overlay
code, based on the starting address of the program.  This information is
"transient" because it's used only when the overlay is being loaded; it
doesn't retain space in the overlay buffer thereafter.


Loading an Overlay

When a call is made to a function or procedure in an overlay that is not
present in memory, the INT $3F instruction in the dispatcher transfers
control to the overlay manager, which does the following:

1.  Finds space for the overlay (perhaps by booting out other overlays).

2.  Loads the required overlay code into memory and updates LoadSegment in the
header.

3.  Changes the INT $3F instructions in the dispatcher to FAR JMPs.  The
code offset word following the INT $3F is used here to create the jump
instruction.

4.  Adjusts the return address of the INT instruction.  If left alone, this
would return to the third byte of the FAR JMP instruction.  It's adjusted
to return to the beginning of the FAR JMP.

5.  Returns to the dispatcher.  This tranfers control to the FAR JMP code,
which in turn jumps to the requested procedure.


Removing an Overlay

Special precautions are taken when an overlaid unit is to be removed from
the overlay buffer.  Without this preparation, when you are about to
return to an overlaid unit that is no longer in the overlay buffer, maybe
because you call one overlay from another, the return address on the stack
would point to the physical address where the CALL instruction USED TO BE.
This clearly won't work.  Something must be done to avoid this situation,
and that's where ReturnInt and ReturnOfs enter the picture.

When a unit is to be moved or removed from the overlay buffer, the overlay
manager traverses the stack, looking for return addresses that point into
that unit.  Since a unit is not likely to be reloaded at the same address it
came from, all return addresses into the unit must be changed.  It does this
by jumping through the BP links on the stack until it meets the zero pushed
by the Turbo RTL startup code.  This is the reason all global routines in an
overlaid unit must be FAR and must set up a stack frame.  In each return
address on the stack, the segment value is changed to point to the static
dispatcher segment.  That way they can be recognized when they are to be
changed again during the reload process.  The most-recently-pushed return
address needs special handling, since it the first one that will be accessed
and will cause the unit to be reloaded.  Its offset is copied into ReturnOfs
to be used later, and the offset part on the stack is set to zero so that it
points to ReturnInt, which contains an INT $3F instruction.  This sets the
stage for a safe landing when it's time to make the return trip.


Reloading the Overlay

When the program reaches the RETF instruction that wants to return to the
unit that was unloaded, control passes to the INT $3F instruction at offset
zero of the dispatcher.  This transfers control to the overlay manager,
which reloads the unit, and then traverses the stack once more, finding the
return addresses having segment values pointing to the static dispatcher
segment, and changing them to the actual physical segment in the overlay
buffer.  There's no need to change the offset parts; an overlay is always
loaded at a paragraph boundary, so the offsets don't change.  The overlay
manager then uses the value from ReturnOfs to create a return address to the
appropriate point in the code.



If you have any additions or corrections for this file, please contact 
Ron Schuster [76666,2322] via EasyPlex on CompuServe.

Contributors:
Kim Kokkonen [76004,2611]
Per Larsen [75470,1320]
Ron Schuster [76666,2322]
Roy Furman [72346,72]

01/04/90  Version 4