@c -*-texinfo-*-
@setfilename ../info/internals
@node GNU Emacs Internals, Standard Errors, Tips, Top
@comment  node-name,  next,  previous,  up
@appendix GNU Emacs Internals

This chapter describes how the runnable Emacs executable is dumped with
the preloaded Lisp libraries in it, how storage is allocated, and some
internal aspects of GNU Emacs that may be of interest to C programmers.

@menu
* Building Emacs::      How to preload Lisp libraries into Emacs.
* Pure Storage::        A kludge to make preloaded Lisp functions sharable.
* Garbage Collection::  Reclaiming space for Lisp objects no longer used.
* Object Internals::    Data formats of buffers, windows, processes.
* Writing Emacs Primitives::   Writing C code for Emacs.
@end menu

@node Building Emacs, Pure Storage, GNU Emacs Internals, GNU Emacs Internals
@appendixsec Building Emacs
@cindex building Emacs
@pindex temacs

  The first step in building Emacs is to compile the C sources.  This
produces a program called @code{temacs}, also called a @dfn{bare impure
Emacs}.  It contains the Emacs Lisp interpreter and I/O routines, but
not the editing commands.

@cindex @file{loadup.el}
  Then, to create a working Emacs editor, issue the command @samp{temacs
-l loadup}.  This directs @code{temacs} to evaluate the Lisp files
specified in the file @file{loadup.el}.  These files set up the normal
Emacs editing environment, resulting in an Emacs which is still impure
but no longer bare.

@pindex xemacs
  It takes long time to load the standard Lisp files.  Luckily, you
don't have to do this each time you run Emacs; @code{temacs} can dump
out an executable program called @code{xemacs} which has these files
preloaded.  @code{xemacs} starts more quickly because it does not need
to load the files.  It is @code{xemacs} that is normally installed under
the name @code{emacs} for users to run.

  To create @code{xemacs}, use the command @samp{temacs -batch -l loadup
dump}.  The purpose of @samp{-batch} here is to prevent @code{temacs}
from trying to initialize any of its data on the terminal; this ensures
that the tables of terminal information are empty in the dumped Emacs.

  When the @code{xemacs} executable is started, it will automatically
load the user's @file{.emacs} file, or the default initialization file
@file{default.el} if the user has none.  With the @file{.emacs} file,
you can produce a version of Emacs that suits you and is not the same as
the version other people use.  With @file{default.el}, you can customize
Emacs for all the users at your site who don't choose to customize it
for themselves.  (For further reflection: why is this different from the
case of the barber who shaves every man who doesn't shave himself?)

  On some systems, dumping does not work.  Then, you must start Emacs
with the @samp{temacs -l loadup} command each time you use it.  This
takes a long time, but since you need to start Emacs once a day at
most---and once a week or less frequently if you never log out---the
extra time is not too severe a problem.

  Before @code{xemacs} is dumped, the documentation strings for
primitive and preloaded functions (and variables) need to be found in
the file where they are stored.  This is done by calling
@code{Snarf-documentation} (@pxref{Accessing Documentation}).  These
strings are omitted from @code{temacs} to save space.
@xref{Documentation Basics}.

@defun dump-emacs to-file from-file
@cindex unexec
  This function dumps the current state of Emacs into an executable file
@var{to-file}.  It takes symbols from @var{from-file} (this is normally
the executable file @file{temacs}).

If you use this function in an Emacs that was already dumped, you must
set @code{command-line-processed} to @code{nil} first for good results.
@xref{Command Line Arguments}.
@end defun

@deffn Command emacs-version
  This function returns a string describing the version of Emacs that is
running.  It is useful to include this string in bug reports.

@example
(emacs-version)
  @result{} "GNU Emacs 18.36.1 of Fri Feb 27 1987 on slug (berkeley-unix)"
@end example

Called interactively, the function prints the same information in the
echo area.
@end deffn

@defvar emacs-build-time
  The value of this variable is the time at which Emacs was built at the
local site.

@example
emacs-build-time
     @result{} "Fri Feb 27 14:55:57 1987"
@end example
@end defvar

@defvar emacs-version
  The value of this variable is the version of Emacs being run.  It is a
string, e.g. @code{"18.36.1"}.
@end defvar

@node Pure Storage, Garbage Collection, Building Emacs, GNU Emacs Internals
@appendixsec Pure Storage
@cindex pure storage

  There are two types of storage in GNU Emacs Lisp for user-created Lisp
objects: @dfn{normal storage} and @dfn{pure storage}.  Normal storage is
where all the new data which is created during an Emacs session is kept;
see the following section for information on normal storage.  Pure
storage is used for certain data in the preloaded standard Lisp files:
data that should never change during actual use of Emacs.

  Pure storage is allocated only while @code{temacs} is loading the
standard preloaded Lisp libraries.  In the file @code{xemacs}, it is
marked as read-only (on operating systems which permit this), so that
the memory space can be shared by all the Emacs jobs running on the
machine at once.  Pure storage is not expandable; a fixed amount is
allocated when Emacs is compiled, and if that is not sufficient for the
preloaded libraries, @code{temacs} crashes.  If that happens, you will
have to increase the compilation parameter @code{PURESIZE} in the file
@file{config.h}.  This normally won't happen unless you try to preload
additional libraries or add features to the standard ones.

@defun purecopy object
  This function makes a copy of @var{object} in pure storage and returns
it.  It copies strings by simply making a new string with the same
characters in pure storage.  It recursively copies the contents of
vectors and cons cells.  It does not make copies of symbols, or any
other objects, but just returns them unchanged.  It signals an error if
asked to copy markers.

This function is used only while Emacs is being built and dumped, and is
called only in the file @file{emacs/lisp/loaddefs.el}.
@end defun

@defvar pure-bytes-used
  The value of this variable is the number of bytes of pure storage
allocated so far.  Typically, in a dumped Emacs, this number is very
close to the total amount of pure storage available---if it were not,
we would preallocate less.
@end defvar

@defvar purify-flag
  This variable determines whether @code{defun} should make a copy of the
function definition in pure storage.  If it is non-@code{nil}, then the
function definition is copied into pure storage.

  This flag is @code{t} while loading all of the basic functions for
building Emacs initially (allowing those functions to be sharable and
non-collectible).  It is set to @code{nil} when Emacs is saved out
as @code{xemacs}.  The flag is set and reset in the C sources.

 You should not change this flag in a running Emacs.
@end defvar

@node Garbage Collection, Writing Emacs Primitives, Pure Storage, GNU Emacs Internals
@appendixsec Garbage Collection
@cindex garbage collector

@cindex memory allocation
  When a program creates a list or the user defines a new function (such
as by loading a library), then that data is placed in normal storage.
If normal storage runs low, then Emacs asks the operating system to
allocate more memory in blocks of 1k bytes.  Each block is used for one
type of Lisp object, so symbols, cons cells, markers, etc. are
segregated in distinct blocks in memory.  (Vectors, buffers and certain
other editing types, which are fairly large, are allocated in individual
blocks, one per object, while strings are packed into blocks of 8k
bytes.)

  It is quite common to use some storage for a while, then release it
by, for example, killing a buffer or deleting the last pointer to an
object.  Emacs provides a @dfn{garbage collector} to reclaim this
abandoned storage.  (This name is traditional, but ``garbage recycler''
might be a more intuitive metaphor for this facility.)

  The garbage collector operates by scanning all the objects that have
been allocated and marking those that are still accessible to Lisp
programs.  To begin with, all the symbols, their values and associated
function definitions, and any data presently on the stack, are
accessible.  Any objects which can be reached indirectly through other
accessible objects are also accessible.

  When this is finished, all inaccessible objects are garbage.  No
matter what the Lisp program or the user does, it is impossible to refer
to them, since there is no longer a way to reach them.  Their
space might as well be reused, since no one will notice.  That is what
the garbage collector arranges to do.

@cindex free list
  Unused cons cells are chained together onto a @dfn{free list} for
future allocation; likewise for symbols and markers.  The accessible
strings are compacted so they are contiguous in memory; then the rest of
the space formerly occupied by strings is made available to the string
creation functions.  Vectors, buffers, windows and other large objects
are individually allocated and freed using @code{malloc}.

@quotation
@b{Common Lisp note:} unlike other Lisps, GNU Emacs Lisp does not
call the garbage collector when the free list is empty.  Instead, it
simply requests the operating system to allocate more storage, and
processing continues until @code{gc-cons-threshold} bytes have been
used.

This means that you can make sure that the garbage collector will not
run during a certain portion of a Lisp program by calling the garbage
collector explicitly just before it (provided that portion of the
program does not use so much space as to force a second garbage
collection).
@end quotation

@deffn Command garbage-collect
  This command runs a garbage collection, and returns information on
the amount of space in use.  (Garbage collection can also occur
spontaneously if you use more than @code{gc-cons-threshold} bytes of
Lisp data since the previous garbage collection.)

  @code{garbage-collect} returns a list containing the following
information:

@example
((@var{used-conses} . @var{free-conses})
 (@var{used-syms} . @var{free-syms})
 (@var{used-markers} . @var{free-markers})
 @var{used-string-chars} 
 @var{used-vector-slots})

(garbage-collect)
     @result{} ((3435 . 2332) (1688 . 0) (57 . 417) 24510 3839)
@end example

Here is a table explaining each element:

@table @var
@item used-conses
The number of cons cells in use.

@item free-conses
The number of cons cells for which space has been obtained from the
operating system, but that are not currently being used.

@item used-syms
The number of symbols in use.

@item free-syms
The number of symbols for which space has been obtained from the
operating system, but that are not currently being used.

@item used-markers
The number of markers in use.

@item free-markers
The number of markers for which space has been obtained from the
operating system, but that are not currently being used.

@item used-string-chars
The total size of all strings, in characters.

@item used-vector-slots
The total number of elements of existing vectors.
@end table
@end deffn

@defopt gc-cons-threshold
  The value of this variable is the number of bytes of storage that must
be allocated for Lisp objects after one garbage collection in order to
request another garbage collection.  A cons cell counts as eight bytes,
a string as one byte per character plus a few bytes of overhead, and so
on.  (Space allocated to the contents of buffers does not count.)  Note
that the new garbage collection does not happen immediately when the
threshold is exhausted, but only the next time the Lisp evaluator is
called.

  The initial threshold value is 100,000.  If you specify a larger
value, garbage collection will happen less often.  This reduces the
amount of time spent garbage collecting, but increases total memory use.
You may want to do this when running a program which creates lots of
Lisp data.

  You can make collections more frequent by specifying a smaller value,
down to 10,000.  A value less than 10,000 will remain in effect only
until the subsequent garbage collection, at which time
@code{garbage-collect} will set the threshold back to 100,000.
@end defopt

@node Writing Emacs Primitives, Object Internals, Garbage Collection, GNU Emacs Internals
@appendixsec Writing Emacs Primitives
@cindex primitive function internals

  Lisp primitives are Lisp functions implemented in C.  The details of
interfacing the C function so that Lisp can call it are handled by a few
C macros.  The only way to really understand how to write new C code is
to read the source, but we can explain some things here.

  An example of a special form is the definition of @code{or}, from
@file{eval.c}.  (An ordinary function would have the same general
appearance.)

@cindex garbage collection protection
@example
DEFUN ("or", For, Sor, 0, UNEVALLED, 0,
  "Eval args until one of them yields non-NIL, then return that value.\n\
The remaining args are not evalled at all.\n\
If all args return NIL, return NIL.")
  (args)
     Lisp_Object args;
@{
  register Lisp_Object val;
  Lisp_Object args_left;
  struct gcpro gcpro1;

  if (NULL(args))
    return Qnil;

  args_left = args;
  GCPRO1 (args_left);

  do
    @{
      val = Feval (Fcar (args_left));
      if (!NULL (val))
        break;
      args_left = Fcdr (args_left);
    @}
  while (!NULL(args_left));

  UNGCPRO;
  return val;
@}
@end example

  Let's start with a precise explanation of the arguments to the
@code{DEFUN} macro:

@enumerate
@item
The first argument is the name of the Lisp symbol to define with this
function; it is @code{or}.

@item
The second argument is the C function name for this function.  This is
the name that is used in C code for calling the function.  The name is,
by convention, @samp{F} prepended to the Lisp name, with all dashes
(@samp{-}) in the Lisp name changed to underscores.  Thus, to call this
function from C code, call @code{For}.  Remember that the arguments must
be of type @code{Lisp_Object}; various macros and functions for creating
values of type @code{Lisp_Object} are declared in the file
@file{lisp.h}.

@item
The third argument is a C variable name to use for a structure that
holds the data for the subr object that represents the function in Lisp.
This structure conveys the Lisp symbol name to the initialization
routine that will create the symbol and store the subr object as its
definition.  By convention, this name is the C function name with
@samp{F} replaced with @samp{S}.

@item
The fourth argument is the minimum number of arguments that the function
requires.  In this case, no arguments are required.

@item
The fifth argument is the maximum number of arguments that the function
accepts.  Alternatively, it can be @code{UNEVALLED}, indicating a
special form that receives unevaluated arguments.  A function with the
equivalent of an @code{&rest} argument would have @code{MANY} in this
position.  Both @code{UNEVALLED} and @code{MANY} are macros.  This
argument must be one of these macros or a number at least as large as
the fourth argument.

@item
The sixth argument is an interactive specification, a string such as
might be used as the argument of @code{interactive} in a Lisp function.
In this case it is 0 (a null pointer), indicating that this function
cannot be called interactively.  A value of @code{""} indicates an
interactive function not taking arguments.

@item
The last argument is the documentation string.  It is written just like
a documentation string for a function defined in Lisp, except you must
write @samp{\n\} at the end of each line.  In particular, the first line
should be a single sentence.
@end enumerate

  After the call to the @code{DEFUN} macro, you must write the list
of argument names that every C function must have, followed by
ordinary C declarations for them.  Normally, all the arguments must
be declared as @code{Lisp_Object}.  If the function has no upper limit
on the number of arguments in Lisp, then in C it receives two arguments:
the number of Lisp arguments, and the address of a block containing their
values.  These have types @code{int} and @code{Lisp_Object *}.

  Within the function @code{For} itself, note the use of the macros
@code{GCPRO1} and @code{UNGCPRO}.  @code{GCPRO1} is used to ``protect''
a variable from garbage collection---to inform the garbage collector that
it must look in that variable and regard its contents as an accessible
object.  This is necessary whenever you call @code{Feval} or anything
that can directly or indirectly call @code{Feval}.  At such a time, any
Lisp object that you intend to refer to again must be protected somehow.
@code{UNGCPRO} cancels the protection of the variables that are
protected in the current function.  It is necessary to do this explicitly.

  For most data types, it suffices to know that one pointer to the
object is protected; as long as the object is not recycled, all pointers
to it remain valid.  This is not so for strings, because the garbage
collector can move them.  When a string is moved, any pointers to it
that the garbage collector does not know about will not be properly
relocated.  Therefore, all pointers to strings must be protected across
any point where garbage collection may be possible.

  The macro @code{GCPRO1} protects just one local variable.  If you
want to protect two, use @code{GCPRO2} instead; repeating @code{GCPRO1}
will not work.  There are also @code{GCPRO3} and @code{GCPRO4}.

  In addition to using these macros, you must declare the local
variables such as @code{gcpro1} which they implicitly use.  If you
protect two variables, with @code{GCPRO2}, you must declare
@code{gcpro1} and @code{gcpro2}, as it uses them both.  Alas, we can't
explain all the tricky details here.

  Defining the C function is not enough; you must also create the
Lisp symbol for the primitive and store a suitable subr object
in its function cell.  This is done by adding code to an initialization
routine.  The code looks like this:

@example
defsubr (&@var{subr-structure-name});
@end example

@noindent
@var{subr-structure-name} is the name you used as the third argument to
@code{DEFUN}.

  If you are adding a primitive to a file that already has Lisp
primitives defined in it, find the function (near the end of the file)
named @code{syms_of_@var{something}}, and add that function call to it.
If the file doesn't have this function, or if you create a new file, add
to it a @code{syms_of_@var{filename}} (e.g., @code{syms_of_myfile}).
Then find the spot in @file{emacs.c} where all of these functions are
called, and add a call to @code{syms_of_@var{filename}} there.

  This function @code{syms_of_@var{filename}} is also the place to
define any C variables which are to be visible as Lisp variables.
@code{DEFVAR_LISP} is used to make a C variable of type
@code{Lisp_Object} visible in Lisp.  @code{DEFVAR_INT} is used to make a
C variable of type @code{int} visible in Lisp with a value that is an
integer.

  Here is another function, with more complicated arguments.  This comes
from the code for the X Window System, and it demonstrates the use of
macros and functions to manipulate Lisp objects.

@example
DEFUN ("coordinates-in-window-p", Fcoordinates_in_window_p,
  Scoordinates_in_window_p, 2, 2,
  "xSpecify coordinate pair: \nXExpression which evals to window: ",
  "Return non-nil if POSITIONS (a list, (SCREEN-X SCREEN-Y)) is in WINDOW.\n\  
  Returned value is list of positions expressed\n\
  relative to window upper left corner.")
  (coordinate, window)
     register Lisp_Object coordinate, window;
@{
  register Lisp_Object xcoord, ycoord;

  if (!CONSP (coordinate)) wrong_type_argument (Qlistp, coordinate);
  CHECK_WINDOW (window, 2);
  xcoord = Fcar (coordinate);
  ycoord = Fcar (Fcdr (coordinate));
  CHECK_NUMBER (xcoord, 0);
  CHECK_NUMBER (ycoord, 1);
  if ((XINT (xcoord) < XINT (XWINDOW (window)->left))
      || (XINT (xcoord) >= (XINT (XWINDOW (window)->left)
                            + XINT (XWINDOW (window)->width))))
    @{
      return Qnil;
    @}
  XFASTINT (xcoord) -= XFASTINT (XWINDOW (window)->left);
  if (XINT (ycoord) == (screen_height - 1))
    return Qnil;
  if ((XINT (ycoord) < XINT (XWINDOW (window)->top))
      || (XINT (ycoord) >= (XINT (XWINDOW (window)->top)
                            + XINT (XWINDOW (window)->height)) - 1))
    @{
      return Qnil;
    @}
  XFASTINT (ycoord) -= XFASTINT (XWINDOW (window)->top);
  return (Fcons (xcoord, Fcons (ycoord, Qnil)));
@}
@end example

  Note that you cannot directly call functions defined in Lisp as, for
example, the primitive function @code{Fcons} is called above.  You must
create the appropriate Lisp form, protect everything from garbage
collection, and @code{Feval} the form, as was done in @code{For} above.

  @file{eval.c} is a very good file to look through for examples;
@file{lisp.h} contains the definitions for some important macros and
functions.

@node Object Internals,  , Writing Emacs Primitives, GNU Emacs Internals
@appendixsec Object Internals
@cindex object internals

  GNU Emacs Lisp manipulates many different types of data.  The actual
data are stored in a heap and the only access that programs have to it is
through pointers.  Pointers are thirty-two bits wide in most
implementations.  Depending on the operating system and type of machine
for which you compile Emacs, twenty-four to twenty-six bits are used to
address the object, and the remaining six to eight bits are used for a
tag that identifies the object's type.

  Because all access to data is through tagged pointers, it is always
possible to determine the type of any object.  This allows variables to
be untyped, and the values assigned to them to be changed without regard
to type.  Function arguments also can be of any type; if you want a
function to accept only a certain type of argument, you must check the
type explicitly using a suitable predicate (@pxref{Type Predicates}).
@cindex type checking internals

@menu
* Buffer Internals::    Components of a buffer structure.
* Window Internals::    Components of a window structure.
* Process Internals::   Components of a process structure.
@end menu

@node Buffer Internals, Window Internals, Object Internals, Object Internals
@appendixsubsec Buffer Internals
@cindex internals, of buffer
@cindex buffer internals

  Buffers contain fields not directly accessible by the Lisp programmer.
We describe them here, naming them by the names used in the C code.
Many are accessible indirectly in Lisp programs via Lisp primitives.

@table @code
@item name
The buffer name is a string which names the buffer.  It is guaranteed to
be unique.  @xref{Buffer Names}.

@item save_modified
This field contains the time when the buffer was last saved, as an integer.
@xref{Buffer Modification}.

@item modtime
This field contains the modification time of the visited file.  It is
set when the file is written or read.  Every time the buffer is written
to the file, this field is compared to the modification time of the
file.  @xref{Buffer Modification}.

@item auto_save_modified
This field contains the time when the buffer was last auto-saved.

@item last_window_start
This field contains the @code{window-start} position in the buffer as of
the last time the buffer was displayed in a window.

@item undodata
This field points to the buffer's undo stack.  @xref{Undo}.

@item syntax_table_v
This field contains the syntax table for the buffer.  @xref{Syntax Tables}.

@item markers
This field contains the chain of all markers that point into the
buffer.  At each deletion or motion of the buffer gap, all of these
markers must be checked and perhaps updated.  @xref{Markers}.

@item backed_up
This field is a flag which tells whether a backup file has been made
for the visited file of this buffer.

@item mark
This field contains the mark for the buffer.  The mark is a marker,
hence it is also included on the list @code{markers}.  @xref{The Mark}.

@item local_var_alist
This field contains the association list containing all of the variables
local in this buffer, and their values.  A copy of this list is returned
by the function @code{buffer-local-variables}.  @xref{Buffer-Local
Variables}.

@item mode_line_format
This field contains a Lisp object which controls how to display the mode
line for this buffer.  @xref{Mode Line Format}.
@end table

@node Window Internals, Process Internals, Buffer Internals, Object Internals
@appendixsubsec Window Internals
@cindex internals, of window
@cindex window internals

  Windows have the following accessible fields:

@table @code
@item height
  The height of the window, measured in lines.

@item width
  The width of the window, measured in columns.

@item buffer
  The buffer which the window is displaying.  This may change often during
the life of the window.

@item start
 The position in the buffer which is the first character to be displayed
in the window.

@item pointm
@cindex window point internals
  This is the value of point in the current buffer when this window is
selected; when it is not selected, it retains its previous value.

@item left
  This is the left-hand edge of the window, measured in columns.  (The
leftmost column on the screen is @w{column 0}.)

@item top
  This is the top edge of the window, measured in lines.  (The top line on
the screen is @w{line 0}.)

@item next
  This is the window that is the next in the chain of siblings.

@item prev
  This is the window that is the previous in the chain of siblings.

@item force_start
  This is a flag which, if non-@code{nil}, says that the window has been
scrolled explicitly by the Lisp program.  At the next redisplay, if
point is off the screen, instead of scrolling the window to show the
text around point, point will be moved to a location that is on the
screen.

@item hscroll
  This is the number of columns that the display in the window is scrolled
horizontally to the left.  Normally, this is 0.

@item use_time
  This is the last time that the window was selected.  This field is used
by @code{get-lru-window}.
@end table

@node Process Internals,  , Window Internals, Object Internals
@appendixsubsec Process Internals
@cindex internals, of process
@cindex process internals

  The fields of a process are:

@table @code
@item name
A string, the name of the process.

@item command
A list containing the command arguments that were used to start this
process.

@item filter
A function used to accept output from the process instead of a buffer,
or @code{nil}.

@item sentinel
A function called whenever the process receives a signal, or @code{nil}.

@item buffer
The associated buffer of the process.

@item pid
An integer, the Unix process @sc{id}.

@item childp
A flag, non-@code{nil} if this is really a child process.
It is @code{nil} for a network connection.

@item flags
A symbol indicating the state of the process.  Possible values include
@code{run}, @code{stop}, @code{closed}, etc.

@item reason
An integer, the Unix signal number that the process received that
caused the process to terminate or stop.  If the process has exited,
then this is the exit code it specified.

@item mark
A marker indicating the position of end of last output from this process
inserted into the buffer.  This is usually the end of the buffer.

@item kill_without_query
A flag, non-@code{nil} meaning this process should not cause
confirmation to be needed if Emacs is killed.
@end table
