
                                                     July 1987

                      The History and Technology
              of TSR Terminate and Stay Resident software

                                  by
                             Steve Gibson
                   InfoWorld TechTalk Columnist and
                  President of Gibson Research Corp.


                            HISTORY BEGINS
----------------------------------------------------------------------
Once upon a time a Big Blue company named IBM decided that it wanted 
to make personal computers too.  For reasons known only to the Gods of 
Boca Raton Florida, IBM decided to use the 8088 microcomputer from 
Intel rather than Motorola's vastly "cleaner" 68000 (as is used in 
Apple's macintosh).  Viewed in retrospect, this has turned out to be a 
ridiculously expensive, i.e. WRONG, decision.  But anyway... IBM 
journeyed to the West to get its software for this new machine.  

IBM first visited the company who was then the acknowledged king of 
operating systems, Digital Research, Inc.  DRI had developed CP/M, the 
defacto standard operating system for the very popular 8080 and Z80 
microcomputer systems.  IBM explained that it needed an operating 
system for the new 8088 microprocessor in their new, blue, personal 
computer and asked if DRI would please provide them with one.  DRI 
said, quite truthfully, that they didn't have one.  Big blue must have 
then asked if they would like to make one, to which DRI must have 
again said: "No thanks." 

IBM shrugged and headed North. . . . . 

Though IBM may have only planned to ask Microsoft for an 8088 BASIC 
language interpreter for their new machine, they found themselves in 
Washington without an operating system.  Microsoft said they would be 
pleased to provide IBM with both an 8088 operating system and the 
BASIC language interpreter.  IBM said "Thank you" and said they needed 
it very soon please. 

But Microsoft didn't have an 8088 operating system either, and being 
so very much smaller then than now, didn't have the resources to build 
one from scratch by IBM's deadline.  So they found an even tinier 
company called Seattle Computer who had built a little unpretentious 
operating system for the 8088 and 8086 micros called 86-DOS.  It was 
also known by some as QDOS.  QDOS was said to stand for Quick and 
Dirty Operating System. 

So Microsoft, then able to meet IBM's 8088 machine timetable, 
delivered both an operating system (albeit one of unassuming parentage 
and questionable virtue) and a BASIC interpreter. 

Everything seemed fine until we all began really using the new 
operating system.  We wanted additional and rather reasonable things 
from MS-DOS and PC-DOS.  Things which had not been designed into 86-
DOS, like printing in the background while the user was doing other 
things.  Or supporting RS-232 serial printers, which required routing 
the printer data out through a serial port instead of the normal 
parallel printer port. 

To meet these reasonable demands Microsoft made a series of small 
changes and additions to the guts of MS-DOS which allowed extra "add-
on" code to be "hung" onto the outside of MS-DOS and also provided for 
limited communication between DOS and the add-on code.  Microsoft's 
idea was that these MS-DOS limitations would be "fixed" by writing 
some new DOS commands (in the above examples PRINT and MODE) which 
would make use of special undocumented aspects of their latest version 
operating system. 

The problem was that MS/PC-DOS, still really old 86-DOS (QDOS) in 
disguise, was never built to support multitasking.  This means that it 
was never meant to serve more than a single master at a time.  A 
command like PRINT, which can continue to operate even AFTER the user 
has begun doing something else, could very easily confuse it, since 
now there could be service requests coming into MS-DOS from two quite 
different places at once. 

Since this began making the entire operating system rather fragile and 
weird, Microsoft's grand plan was to retain all these special hooks 
and communications paths as a company secret.  They figured that as 
long as they were the only ones to use these new MS-DOS secrets things 
wouldn't get out of hand. 

But Microsoft must have gravely underestimated the cleverness and 
determination of the development community.  When the MODE command 
proudly stated: "Resident portion installed" developer's eyebrows shot 
up!  RESIDENT portion?  Hmmmmmm.  And if the PRINT command could 
access the disk for file printing when we were messing around with a 
spreadsheet, why couldn't we build a little resident notepad or two, 
too? 

But EVEN after these new resident goodies began appearing in 
increasing numbers, Microsoft refused to "disclose" anything 
whatsoever.  They said they "couldn't" document these things or they'd 
be forced to make all FUTURE versions of the operating system support 
them in upwardly compatible fashion.  Since these were awkward 
"kludge" solutions to the real need for a true multitasking operating 
system, Microsoft's delicate condition can be readily understood. 

However, developers sensing massive market opportunities were not to 
be stopped.  Debugging software which allowed programmers to peer 
inside the still working machine sold like wildfire.  Using these 
debuggers (like PERISCOPE, my personal favorite), developers probed 
into the very heart of Microsoft's own PRINT and MODE commands, 
unscrambling and deciphering the undocumented secrets of their 
operation.  Using the same secrets Microsoft's own developers had 
used, they learned how to give their OWN programs similar 
capabilities. 

In an old episode of Star Trek, Mr. Spock said, after capturing the 
Romulan's cloaking device, "...military secrets are the most fleeting 
of all, Commander."  This applies equally well to technological 
secrets in general, and as we've seen, to Resident Software in 
particular! 

It's unfortunate that Microsoft did not have time back then in the 
beginning to build their own DOS from scratch.  They might have done 
it right.  But we're stuck with what we have today, and making the 
most of it is certainly our best strategy. 

It remains unfortunate, though certainly understandable, that 
Microsoft isn't willing to disclose the details of MS-DOS's resident 
software hooks and techniques.  This policy keeps such knowledge quite 
secret, since even the third-party developers are reluctant to 
disclose their own hard-won "insider's" knowledge and proprietary 
techniques for forcing reliable multitasking behavior from good ol' 
single tasking DOS.  It's quite interesting to speculate that perhaps 
today even Microsoft does not KNOW how to write MS-DOS multitasking 
applications as well and robustly as several of the larger commercial 
resident software publishers!  The tricks required to make DOS 
multitask are real nasty stomach-turners. 

So... what are these tricks and techniques?  How DO resident programs 
"pop up over" applications at the simple press of a "hot key"? 



                  THE TECHNOLOGY OF RESIDENT SOFTWARE
----------------------------------------------------------------------

Since we've now know how we got to wherever it is we are, let's answer 
the those questions posed above.... 

Two things are required for the functioning of TSR technology:  First, 
the TSR program needs to somehow "hang around" well after any other 
program has started to run.  Secondly, it has to "know" what's going 
on in the system to be able to jump in and help out when called upon 
for action.  It also needs some way of re-assuming control of the 
machine when the time comes for it to take action. 

The requirement of continuing RAM residency is satisfied by a service 
provided by the operating system from which TSR's get their name: 
Terminate and Stay Resident (TSR). 

The main operating RAM memory of an IBM compatible PC is a contiguous 
block beginning at address zero (called the bottom of RAM memory) and 
extending "upward" for a maximum of 640K bytes (called the top of 
memory).  Note that 640K is actually 655,360 bytes since "Ks" in 
computerdom are really 1024 bytes, not 1000.  Since the very bottom of 
this large contiguous block of RAM is reserved for system housekeeping 
purposes, DOS is "booted" by copying it from disk into RAM just 
"above" this low-RAM region. 

Once DOS has been loaded, all the space "above" it, to the top of RAM, 
is available for application program loading and use.  We'll call this 
location just above DOS the LOAD POINT for application software.  So, 
when a program is started, it is copied from disk into RAM starting at 
this LOAD POINT, continuing upward until the entire program has been 
loaded.  The memory available, and allocated, to the program for its 
own data storage and manipulation extends from the end of itself to 
the physical top of the RAM memory. 

Most standard programs word processors, spreadsheets, or databases 
have "control" of the computer until the user tells the program to 
terminate.  After performing whatever cleanup and file saving might be 
required, the program simply informs DOS that it is all finished by 
saying the equivalent of "I'm all finished now, terminate me please". 



Responding to this "Terminate" request from a program, DOS simply 
"forgets" all about the program and whatever data it might have had in 
RAM.  It displays the command line prompt and awaits another command.  
The next program to run is loaded at the same LOAD POINT as before, 
thus overwriting whatever terminated program was just running, and 
implicitly making the same amount of memory available to this next 
program. 

Now consider the special case of the RESIDENT PROGRAM... 

It begins innocently, just like any other application program, when 
its name is typed at the DOS prompt.  DOS copies it from disk into the 
user's RAM memory starting at the same LOAD POINT as any prior 
application software, implicitly leaving the rest of RAM memory for 
the program's own use until done, then hands control of the computer 
system over to this program. 

Now things start getting interesting! 

The resident program will generally say hello to the user announcing 
its intention to hang around for the duration.  Then it does a few 
"non-standard things" to the system.  These are the very things 
MICROSOFT wanted to keep all to itself.  These special "things" keep 
the program from being ex-communicated from system activities after it 
terminates.  Then, rather than simply saying "terminate me" to DOS, it 
says: "I'm xxx bytes long.  Now terminate me but DON'T OVERWRITE ME."  
Which is to say: Terminate me, but let me Stay Resident. 

Upon receiving this request, DOS takes control from the program, moves 
the software LOAD POINT UPWARD by those xxx bytes, and only THEN 
returns the DOS prompt to the user.  Since the LOAD POINT has been 
moved up by "xxx" bytes, all subsequent programs will be loaded AFTER 
the programs which ASKED to remain resident, thus leaving it 
undisturbed in the system's memory! 

Additionally, since it's the resident program which tells DOS how big 
it is, it is able to declare itself to be ANY LENGTH IT CHOOSES!  This 
is how programs like SideKick, which are only 39K bytes long while on 
the disk, can have a 50K notepad file and take up 89K bytes while in 
RAM!  It effectively LIES to DOS about how big it is, so DOS 
obligingly leaves it space for itself and for however much working 
memory it wants!  This is the way resident programs are able to eat up 
so much of our main RAM memory!  When this process is repeated many 
time (as is frequently the case) a resident program "stack" is created 
in main RAM memory. 

So, we've now seen how programs are able to remain in memory after 
being "officially" terminated.  The final question is:  How are these 
"terminated" programs able to remain "aware" of the activities around 
them?... and how are they able to regain control of the computer 
whenever they see fit to do so?  In other words, how can a notepad or 
other utility "pop up" ??  To answer these last questions let's step 
back for a moment, reviewing a bit about how computer's are 
controlled: 






A computer's actions are directed by its reading of instructions which 
are stored in its main memory.  These instructions may be temporary, 
as in the case of programs loaded from disk into RAM, or permanent, as 
in the case of instructions permanently wired into ROM chips.  In 
either case, each instruction has a unique address which is just the 
number of the byte it occupies in the machine's memory. 

We cause the computer to perform a given task by having it read a 
series of instructions which detail the task.  In a sense we CALL upon 
those instructions whenever we need to exert their intended function.  
Such a "CALLED" instruction sequence generally has a clever 
instruction called RETURN which is able to send the computer back to 
wherever it was called from! 

This notion of FUNCTIONS which are being performed by instruction 
sequences called upon in memory, and which will return to us when 
finished, is very powerful.  IBM compatible computers come equipped 
with a BIOS ROM which contains a host of very useful general purpose 
functions which return control to their caller as soon as they have 
performed their designated duty.  This is EXACTLY what that often 
mentioned BIOS ROM is all about!  B.I.O.S. stands for "Basic Input 
Output System", and consists of a collection of standard functions for 
reading and writing to the system keyboard, screen, mass storage, and 
many other standard devices. 

Naturally, we must know the exact address of these "functional" 
services in order to CALL them into service.  By convention, the 
starting address of every built-in function in the IBM PC is located 
in a predefined table of addresses located at the very start of the 
computer's RAM memory.  Every function is numbered simply by the 
location of its address in this table.  This means that in order to 
call for any built-in IBM PC function we only need to know its 
standard function number - from this we're able to calculate the 
location in the table where that function's starting address can be 
found. 

There's even a clever computer instruction which lets you directly 
call upon a function whose ADDRESS is stored at a specific location in 
this table.  So in effect you can simply ask the computer to call 
"standard function number 7".  In response to this function request 
the computer will look into the 7th slot in the table for the address 
of function number 7, and go there directly, returning to you when the 
function has been performed.  In computer lingo, a place which 
contains the address of something else is a POINTER or a VECTOR. 

Even the electrical hardware of the computer is able to call upon 
these standard functions.  In fact many of the standard functions in 
the ROM BIOS are designed to operate hand-in-hand with the PC's 
hardware.  For example, every time a keyboard key is either pressed or 
released the standard function number 9, whose address is naturally in 
location 9 of the Address Table, is called upon by the hardware.  This 
function is called EVEN IF THE SOFTWARE WAS BUSY DOING SOMETHING ELSE 
AT THE TIME! 

This ability of the hardware to INTERRUPT whatever the software might 
be doing at the time is called a "hardware interrupt".  Essentially 
the keyboard hardware says to the processor: 



"Hey there CPU, hold on a second, remember where you are and what you 
are doing, now go perform the function whose address is in slot number 
9, and when that function returns to you, go back to whatever is was 
you were doing when I bugged you just like nothing ever happened." 

This is EXACTLY how we are able to type ahead of our computers.  Even 
if our spreadsheet is lost in thought recalculating our tax refund, 
we're able to type ahead of it!  Each depression and release of a key 
steals a fraction of a moment, a slice of time, away from the 
spreadsheet's recalc.  The function invoked by the keyboard's hardware 
interrupt places the key into a type-ahead buffer ... which the 
spreadsheet will read from as soon as its good and ready ... then 
returns control to the recalc as if nothing at all had happened. 

Now imagine what would happen if some program (like our resident one) 
were to point the POINTER stored in the 9th table location to 
somewhere else, such as to ITSELF, prior to issuing the Terminate and 
Stay Resident (TSR) request to DOS! 

THEN the hardware interrupt occurring at every keyboard action would 
turn control over to THIS TSR'd RESIDENT PROGRAM instead of to the 
normal built-in keyboard handling function!  If the resident program 
wasn't "interested" in that particular keyboard action or keystroke, 
it could simply pass control along to the place where the POINTER HAD 
BEEN POINTING BEFORE just as if it hadn't been evesdropping at all!  
However, if the key action happened to be THIS resident program's 
assigned "HOT KEY", it could retain control of the system as long as 
it wished, before passing it along as if nothing at all had happened! 

That's all there is to it! 

As we've seen, the clever application of these various tricks and 
techniques can significantly expand the power and flexibility of MS-
DOS and PC-DOS based personal computers. 

There are those who feel that resident software is a makeshift 
temporary solution which is not long for this world.  They say that 
TSR programs will be eliminated by the presence of full multitasking 
operating systems.  This author could not disagree more.  I believe 
TSR programs will be here for the duration as necessary and viable 
add-ons to otherwise incomplete solutions. 

So there you have it.  The history and technology of TSR Resident 
Software in a nutshell.  I hope this gives you a better feel for 
what's going on under the hood of your machine! 

Thanks for listening ... and REMEMBER, if you would like to receive 
notification of other free information, software, and our future TSR 
and utility products, simply pickup the phone or drop us a note or a 
card.  We'd be pleased to keep you up to date. 

------------------------------------------------------------------------
GIBSON RESEARCH CORPORATION, BOX 6024, IRVINE, CA  92716, (714) 854-1520
------------------------------------------------------------------------

Again, Thanks!

Steve Gibson.
                          <<< END OF FILE >>>

