Clipper Support Bulletin #3
Clipper 5.0 runtime memory management

Copyright (c) 1991, 1992 Nantucket Corporation.  All rights reserved.


Version:  Clipper 5.0, revisions 1.00 - 1.03
Date:     4th October, 1990
Revised:  21st May, 1992
Status:   Active

================================================================================

This Support Bulletin covers the following topics:

   1. Virtual Memory Manager (VMM)
       1.1  Use of Lim 3.2 Expanded Memory (EMM)
       1.2  Disk Swapping
       1.3  Swap Space
   2. Object Memory (SVOS)
   3. Dynamic Overlaying
       3.1  Configuration
   4. Data File Buffering
   5. Segment Locking
   6. Fixed-Memory Allocator
   7. MEMVAR Tables
   8. Symbol Tables
   9. The Extend System
       9.1  Configuration

Note: this document makes several references to the CLIPPER environment
variable.  For general information on how to configure the runtime environment,
refer to the Clipper 5.0 "Programming and Utilities Guide."

================================================================================
1. Virtual Memory Manager (VMM)
   
   "Virtual memory" is a generic term for hardware or software
   techniques that allow a limited amount of "real" memory to
   emulate a much larger "virtual" memory.
   
   The Clipper 5.0 Virtual Memory Manager (VMM) is implemented in
   software.  The system runs in real mode on any 8086-compatible
   processor; it does not directly use the protected mode virtual
   memory capabilities of the Intel 286 and 386 processors.
   
   The VMM is a "segmented" memory manager.  Virtual memory is
   allocated in segments, each of which may contain from 1K to 64K
   of data.  Initially, the VMM works somewhat like a normal memory
   allocator (although it has an extra advantage: most VM segments
   are movable; the VMM can reorganize them to get maximum
   utilization of real memory).  Unlike a normal memory allocator,
   however, the VMM does not "run out" when the available real
   memory can no longer contain all of the allocated segments.
   Instead, one or more of the least recently used (LRU) segments
   are "swapped out" into secondary storage (or otherwise removed
   from real memory, see below) to make room for new segments.
   Later, if data in a swapped segment must be accessed, it is
   swapped back in, displacing segments that are not currently
   needed.
   
   The maximum virtual address space of the Clipper 5.0 VMM is
   64MB.
   
   ----------------------------------------------------------------------------
   1.1  Use of Lim 3.2 Expanded Memory (EMM)
   
   By default, the Clipper 5.0 VMM uses LIM 3.2 Expanded Memory
   (EMM) as secondary storage.  EMM provides fast access to swapped
   segments.  Up to 8MB of EMM can be used.
   
   Configuration
   -------------
   EMM usage can be controlled using the E setting in the CLIPPER
   environment variable.  Valid settings are from 0 to 8192 (units
   of 1K).  A setting of 0 disables the use of EMM altogether.
   (Note: disabling EMM is not recommended under Clipper 5.0.
   Although EMM is not required, it allows increased performance
   for most applications.  In a few cases, limiting EMM usage may
   be desirable--see the note below under Disk Buffering System.)
   
   Example:  SET CLIPPER=E2048
   
   Some disk caching programs use EMM to cache disk sectors.
   Certain cache programs may cause conflicts with the Clipper 5.0
   VMM system if they assume that application programs never use
   EMM.  The BADCACHE setting in the CLIPPER environment variable
   causes the VMM to preserve and restore the state of the EMM
   "page frame" before and after every EMM access (the EMM page
   frame is an area in real address space through which EMM data is
   accessed).  This should make Clipper's use of EMM undetectable
   to any other process using EMM.  (Note: on some EMM systems the
   BADCACHE setting may adversely affect VMM performance.  It
   should only be used if you experience disk or file corruption
   because of a conflict with a disk cache or other resident
   software.)
   
   Example:  SET CLIPPER=BADCACHE
   
   ----------------------------------------------------------------------------
   1.2  Disk Swapping
   
   If insufficient EMM is available to contain swapped segments, a
   temporary disk file (the "swap file") is created to hold them.
   By default, the VMM will use a maximum of 8MB of disk space for
   the swap file.  Disk space is used only as needed.
   
   Configuration
   -------------
   The maximum size of the disk swap file can be controlled using
   the SWAPK setting in the CLIPPER environment variable.  Valid
   settings are from 256 to 64000 (256K to 64MB).  The SWAPK
   setting has no effect on VMM swapping activity; it merely serves
   to establish an upper limit on the size of the swap file.  If
   the limit is exceeded, the VMM will terminate the application.
   
   Example:  SET CLIPPER=SWAPK:16000
   
   The disk drive and directory where the VMM swap file is created
   can be controlled using the SWAPPATH setting in the CLIPPER
   environment variable.  By default, the swap file is created in
   the current DOS directory.
   
   Example:  SET CLIPPER=SWAPPATH:"C:\SWAP"
   
   ----------------------------------------------------------------------------
   1.3  Swap Space
   
   The term "swap space" refers to the region of real memory where
   the VMM loads virtual memory segments.
   
   Configuration
   -------------
   The VMM allocates the swap space from DOS at startup.  By
   default, all available DOS memory is used for swap space.  Real
   memory usage can be restricted using the X setting of the
   CLIPPER environment variable.  The setting specifies an amount
   of DOS memory (in 1K units) which is to be excluded from the
   VMM.  Valid settings are from 0 to 256.  The X setting is
   provided as a way to reserve DOS memory for other uses (e.g.
   allocation by memory-resident software).  In general, its use is
   undesirable because limiting the swap space decreases VMM
   performance.
   
   Example:  SET CLIPPER=X8
   
========================================================================
2. Object Memory (SVOS)
   
   An "object memory" is a special type of memory manager designed to
   manage complex data values such as character strings and arrays.
   
   The Clipper 5.0 object memory is called the Segmented Virtual
   Object Store (SVOS).  SVOS uses virtual memory to store data
   values, including character strings, arrays, and dynamically
   created (macro-compiled) code blocks.
   
   SVOS provides two important functions beyond the basic
   capabilities offered by the VMM:
   
   1. Memory compaction: stored values are automatically compacted on
   an ongoing basis.  This eliminates fragmentation of the virtual
   memory and reduces swapping.
   
   2. Garbage collection: Some Clipper 5.0 values (e.g. arrays) may
   be referred to by several program variables (or array elements) at
   the same time.  The garbage collection algorithms automatically
   reclaim space occupied by "unreachable" values (values which are
   no longer accessible through any variable or array).
   
   3. To minimize delays associated with memory compaction and
   garbage collection, the SVOS algorithms are dynamically adjusted
   based on the type, size, and persistence of data being stored, the
   amount of memory available (real and EMM), and the overall
   performance of the system.  The algorithms attempt to manage
   memory in such a way that the VMM can swap out infrequently used
   data, freeing up real address space for time-critical operations.
   
   The dynamic nature of the SVOS algorithms makes it difficult to
   precisely state its maximum capacity.  The worst case capacity is
   in excess of 1MB.  The theoretical maximum is in excess of 16MB.
   
========================================================================
3. Dynamic Overlaying
   
   Clipper 5.0, using the supplied special version of .RTLink,
   manages compiled Clipper code using a technique called dynamic
   overlaying.  This technique eliminates the need for complicated
   overlay structures and allows flexible runtime management of
   compiled code.
   
   During linking, .RTLink breaks compiled Clipper modules into fixed-
   size "pages."  These pages are stored either in the executable
   file or in separate overlay files.  The paging architecture
   eliminates restrictions and memory calculations based on the size
   of compiled functions or modules.  Large modules are broken into
   multiple pages; small functions are grouped together in a single
   page.
   
   At execution time, the dynamic overlay manager loads pages based
   on information embedded in the .EXE by the linker.  The dynamic
   pages are loaded into VM segments, allowing the VMM to manage the
   overlay pages on a competetive basis with other uses of memory.
   The paging architecture allows the system to discard low-use
   sections of code even if the code is associated with a pending
   active function; overlays are not required to "nest" in parallel
   with function activations.  Code pages that are being heavily used
   are maintained in memory by the VMM's LRU swapping policy.
   
   When possible, the VMM will place dynamic overlay pages in EMM,
   reducing overlay reads.  Overlay pages are never written to the
   VMM disk swap file, however.  If a VM segment containing an
   overlay page is to be removed from memory altogether, it is simply
   discarded.  If it is needed subsequently, it is re-read from the
   overlay file.
   
   ----------------------------------------------------------------------------
   3.1  Configuration
   
   In addition to virtual memory, the dynamic overlay manager uses a
   dedicated area of real memory to cache the most active dynamic
   overlay pages.  The default size of this area ranges from 4K to
   16K, depending on the amount of real memory available at startup.
   For low memory situations, the size of this area can be restricted
   using the DYNK setting of the CLIPPER environment variable.  Valid
   values are from 4 to 63.  (Note: restricting the size of this area
   may reduce execution speed.)
   
   Example:  SET CLIPPER=DYNK:4
   
   The dynamic overlay manager will hold overlay pages in virtual
   memory only as long as it can maintain an active file handle for
   the disk file containing the pages.  If the file is closed, the
   pages are discarded from virtual memory.  The maximum number of
   file handles used by the dynamic overlay manager can be controlled
   using the DYNF setting of the CLIPPER environment variable.  Valid
   settings are from 1 to 8.  The default setting is 2.  For
   applications which use many separate overlay files, increasing
   this setting can improve performance.
   
   Example:  SET CLIPPER=DYNF:4
   
========================================================================
4. Data File Buffering
   
   The Clipper 5.0 database drivers use virtual memory segments to
   contain database and index file buffers.  This allows the VMM to
   manage these buffers on a competetive basis with other uses of
   memory.  As with dynamic overlay segments, the VMM will swap
   buffer segments into EMM but not to the disk swap file.  If a VM
   segment containing a disk buffer is to be removed from memory
   altogether, it is written back to the appropriate data file.
   
   The file buffering system allocates buffer management tables in
   fixed memory (see FIXED MEMORY ALLOCATOR below).  The actual
   amount used depends on the combined total of real memory and EMM
   available at startup.  The tables use 14 bytes per 1K of real or
   Expanded memory (the tables never exceed 8K, however).  Because
   these tables are allocated in fixed memory, they have an effect on
   the amount of swap space available to the VMM.  In some low-memory
   situations, reducing the amount of available EMM may help to avoid
   "swap space exhausted" errors. (Note: reducing EMM availability
   may seriously degrade VMM performance in very low memory
   situations due to increased disk swapping--see Use of Lim 3.2
   Expanded Memory above.)
   
========================================================================
5. Segment Locking
   
   From time to time, certain VM segments are "locked" by a
   subsystem.  This prevents the VMM from moving or swapping those
   segments until the subsystem unlocks them.  This capability is
   used sparingly to keep the swap space from becoming cluttered with
   locked segments.
   
========================================================================
6. Fixed-Memory Allocator
   
   Clipper 5.0 subsystems use virtual memory to contain code or data
   that needs to be dynamically managed.  Some allocations are not
   appropriate for VM (e.g. system tables which must remain in the
   same location throughout execution).  A C-style fixed memory
   allocator is included for this purpose.  The fixed memory system
   allocates one or more VM segments and permanently locks them in
   real memory.  Space within these segments is then parceled out as
   needed.  Since fixed segments are never unlocked, the VMM locates
   them at the low end of the swap space to prevent them from
   interfering with other operations.
   
========================================================================
7. MEMVAR Tables
   
   Clipper 5.0 offers several different storage classes for program
   variables.  Local and static variables are stored in a dedicated
   area of real memory.  Private and public (MEMVAR) variables are
   stored in VM segments.
   
   Since MEMVAR variables are created and destroyed dynamically
   during execution, the associated VM segments grow dynamically
   depending on how many MEMVAR variables are in use.  For
   performance reasons, these segments remain locked during most
   operations (they are unlocked during RUN commands and certain
   other memory intensive operations).  In low memory situations, the
   MEMVAR tables may have a significant effect on the amount of
   available swap space (each MEMVAR variable uses 20 bytes in the
   MEMVAR segments, which grow in increments of 1K or more).
   
   MEMVAR variables also require symbol information to be maintained
   at runtime (see below).  Converting private and public variables
   to local and static variables may reduce memory requirements for
   some applications.
   
========================================================================
8. Symbol Tables
   
   Clipper maintains symbol information at runtime in order to
   support symbolic references to variables and functions.  Some
   symbol information is embedded in the .EXE file (the Clipper 5.0
   version of .RTLink eliminates duplication among symbols from
   compiled Clipper modules).  Other symbols are created dynamically
   at runtime.
   
   Memory for symbol tables and related information is allocated
   using the fixed-memory allocator.  The symbols are allocated in
   blocks and do not fragment memory.  However, excessive dynamic
   symbol creation (e.g. large numbers of MEMVAR variables created
   via macros) may cause the symbol tables to grow, reducing swap
   space.
   
========================================================================
9. The Extend System
   
   The Extend system is a set of functions that are used by C and
   assembly language programs to interface with Clipper programs.
   The Extend system includes two memory allocation functions,
   _xalloc() and _xgrab(), which allow C programs to allocate memory
   from Clipper.  These functions call the fixed-memory allocator
   (see above).
   
   The Extend system also allows C programs to gain access to Clipper
   data values.  The Extend system _parc() function, used to inspect
   character values passed from Clipper, is defined as returning an
   address in real memory.  Since character values are stored in
   virtual memory under SVOS (see above), the Extend system
   automatically locks the appropriate VM segments in response to
   _parc() requests.  The segments are automatically unlocked when
   the C function returns control to Clipper (they may also be
   unlocked manually from C, see below).
   
   A conflict can occur if a C function locks a VM segment via
   _parc() and then attempts to allocate a large amount of memory
   using the Extend allocators.  If the segment is locked near the
   low end of the swap space, the fixed allocator may be unable to
   lock or expand a fixed segment to satisfy the allocation request.
   If this occurs, a new fixed segment is created and locked higher
   in memory.  This reduces memory efficiency because the VMM is not
   allowed to move fixed segments (in severe cases, the VMM may issue
   a "swap space exhausted" error because the swap space has become
   too cluttered with fixed segments).  To prevent this, the VMM
   maintains a "safety zone" at the low end of the swap space.
   _parc() requests are prevented from locking segments within this
   zone.
   
   ----------------------------------------------------------------------------
   9.1  Configuration
   
   The size of the Extend "safety zone" can be adjusted using the
   EXTHEAP setting of the CLIPPER environment variable.  The default
   setting is 8K.  Valid settings are from 2 to 256 (units of 1K).
   Increasing the safety zone allows C programs to allocate large
   amounts of memory without creating "holes" in the swap space.
   (Note: the default EXTHEAP setting should be sufficient for most
   applications; increasing the value too much may cause the VMM to
   fail when trying to lock a segment in response to a _parc()
   request.)
   
   ----------------------------------------------------------------------------
   9.2  New Extend Function
   
   The Clipper 5.0 VMM defines an entry point called _xunlock().
   This function can be called from C or assembly language programs
   to unlock SVOS segments when large amounts of memory are to be
   allocated after _parc() has been called.  CAUTION: calling
   _xunlock() causes any pointers returned from previous _parc()
   calls to become invalid.  To access a Clipper character value
   after calling _xunlock(), you must make another call to _parc() to
   get a new pointer.
   
   Nantucket will be publishing an API (Application Program
   Interface) for the Clipper 5.0 VMM so that C and Assembly language
   programs can allocate virtual memory directly, reducing demands on
   swap space.
   

                              *  *  *
