








     The Interface Between C News And The Outside World


                       Henry Spencer

                      Dept. of Zoology
                   University of Toronto





_I_n_t_r_o _a_n_d _G_e_n_e_r_a_l_i_t_i_e_s

     C News relates to the ``outside world'', the system  it
is  installed  on,  in  a  number  of  ways.   This document
attempts to enumerate them and explain what goes on.

     In general, C News attempts to rely  only  on  ``common
basic  Unix'', and in particular it is not particularly BSD-
specific or System-V-specific.  Specifically,  it  makes  no
use   of  ornate  locking  mechanisms,  silly  interprocess-
communication  schemes,  peculiar   networking   primitives,
extensions to C, or other annoyances.

_D_i_r_e_c_t_o_r_i_e_s _A_n_d _U_s_e_r_i_d_s

     Most  of   the   components   of   C   News   live   in
/_u_s_r/_l_i_b/_n_e_w_s_b_i_n,  with  control  files in /_u_s_r/_l_i_b/_n_e_w_s and
spooling areas (for current news, inbound traffic, and  out-
bound  traffic) in /_u_s_r/_s_p_o_o_l/_n_e_w_s.  See the document _D_i_r_e_c_-
_t_o_r_y _L_a_y_o_u_t _a_n_d _P_A_T_H _i_n _C _N_e_w_s for elaboration on the struc-
ture,  _F_i_l_e_s _i_n /_u_s_r/_l_i_b/_n_e_w_s (_a_k_a _N_E_W_S_C_T_L) for a summary of
control files, and _C_o_n_f_i_g_u_r_a_t_i_o_n _M_e_c_h_a_n_i_s_m_s _i_n  _C  _N_e_w_s  for
how to change the locations of the directories.

     There is  an  extensive  subdirectory  structure  under
/_u_s_r/_l_i_b/_n_e_w_s_b_i_n,  with only a few heavily-used utilities in
the top-level directory.  In  the  following,  programs  not
explicitly  described as going elsewhere are all under there
somewhere.

     C News's spool areas and major control files need to be
owned by a specific userid, normally _n_e_w_s.  (We believe that
nothing except some of the Makefiles knows  this  name.)  We
suggest that this userid should not own anything else on the
system.  The programs in /_u_s_r/_l_i_b/_n_e_w_s_b_i_n can  be  owned  by
_b_i_n  (or  anyone  else)  except  for  those  that need to be
setuid.  (On our systems the non-setuid ones  are  owned  by
_b_i_n.)




                       3 Feb 1991





                           - 2 -


_U_n_i_x _U_t_i_l_i_t_i_e_s

     C News requires a fairly complete Unix  or  equivalent.
(We  take  no position on whether 4BSD, or System V, is Unix
or not; our private opinion is that neither  truly  deserves
the name any more, although we occasionally change our minds
about which is less deserving.) (Note also that ``Unix''  is
used here as an abbreviation for The UNIX Operating System8r9,
a registered trademark of AT&T; we acquired our  bad  habits
and  incorrect  capitalization from Unix [sic] documentation
supplied by the Bell System in the mid-1970s.)

     In particular, C News relies heavily  on  shell  files.
``Shell'' here means, of course, the standard shell, written
by Steve Bourne.  If your /_b_i_n/_s_h is not a Bourne  shell  or
_v_e_r_y  good  imitation, you're in trouble.  Note, in particu-
lar, that some old versions of the Korn  shell  differ  from
the  Bourne  shell  in  some  important  details  of quoting
behavior, and we _k_n_o_w this causes problems with C News.  You
need a standard shell.

     To the best of our ability, all our shell  files  begin
with  ``#! /bin/sh''.   As  far as we know, nothing actually
relies on being able to _e_x_e_c  a  shell  file;  that  is,  if
``#! /bin/sh''  is just a comment to your shell, that should
be okay.  If your shell misinterprets it as a request to run
the  C  shell, however, you're in trouble.  Running the fol-
lowing might help:

        for f in `find . -type f -print`
        do
                if test " `sed 1q $f`" = "#! /bin/sh"
                then
                        ed $f <<'!'
        1s/#!/: use/
        w
        !
                fi
        done

If your shell doesn't know  about  ``#''  comments  at  all,
again you're in trouble.  We hope that no modern shell makes
either of these mistakes.

     We believe that none of our stuff relies on  relatively
modern  shell  features  like shell functions or _g_e_t_o_p_t_s (as
distinct from _g_e_t_o_p_t).  If _t_e_s_t and _e_c_h_o  are  not  built-in
commands  in  your  shell, efficiency will suffer but every-
thing should still run.  It's possible that  a  few  obscure
things  will break if your shell does not support ``['' as a
synonym for _t_e_s_t, although we try to avoid this usage.

     Within the shell files, C News makes  heavy  use  of  a
wide  variety  of  Unix utilities, notably _s_e_d and _a_w_k.  Any



                       3 Feb 1991





                           - 3 -


_a_w_k should do; in particular, nothing needs the ``new  _a_w_k''
(we  don't run it yet).  Although we have tried to avoid it,
it is possible that some things depend  on  _a_w_k  recognizing
``\t'' inside strings, which very old _a_w_ks didn't.

     If your Unix does not put all standard Unix programs in
the  standard directories-/_b_i_n and /_u_s_r/_b_i_n-you will need to
modify C News's default PATH  to  include  whatever  bizarre
directories  are involved.  See the _C_o_n_f_i_g_u_r_a_t_i_o_n _M_e_c_h_a_n_i_s_m_s
document for details.  In particular, for some  insane  rea-
son,  4BSD  relocated  a  few  standard utilities like _w_c to
/_u_s_r/_u_c_b, which causes trouble (and not just  to  C  News!).
We simply put some links in /_u_s_r/_b_i_n to fix this on our sys-
tems, and this is what we  recommend  you  do,  but  if  you
can't, you'll have to change the PATH.

     Several parts of C News rely on being able to send mail
to  an  administrative  account  (the  name  of which can be
chosen; see _C_o_n_f_i_g_u_r_a_t_i_o_n _M_e_c_h_a_n_i_s_m_s).   The  assumption  is
that  a  message,  without  any RFC822 headers, can be piped
into /_b_i_n/_m_a_i_l (or whatever  _m_a_i_l  is  first  in  the  PATH)
invoked  with  a single argument which is the addressee, and
be delivered to the  addressee's  mailbox  intact,  possibly
embellished  with  headers.   That  is, with news's standard
PATH, if

        echo "relaynews: hi there Joe" | mail joe

does not result in the message ``relaynews: hi  there  Joe''
arriving  in  the  mailbox  ``joe'', you're going to have to
fake it.  (Note that some BSD mailers run into trouble  with
the  colon  in  the  example,  interpreting such a line as a
header.) See _D_i_r_e_c_t_o_r_y _L_a_y_o_u_t for insight on where to put  a
fake  _m_a_i_l so that C News components will use it rather than
/_b_i_n/_m_a_i_l.

     Similarly, several parts of C News rely on  being  able
to  invoke  _h_o_s_t_n_a_m_e without arguments, and have it return a
single word which is the name of the CPU the code is running
on.   Again,  you'll have to fake it if you don't have _h_o_s_t_-
_n_a_m_e or if it outputs something strange.

     Input reception and output batching both  want  to  use
the  _c_o_m_p_r_e_s_s  data-compression  program.  _C_o_m_p_r_e_s_s has been
published on Usenet and is shipped with  many  Unix  systems
these  days, but if you don't have it, we recommend that you
get  it  from  your  neighbors  or   the   _c_o_m_p._s_o_u_r_c_e_s._u_n_i_x
archives.   It  is  possible  to configure C News so that it
doesn't use _c_o_m_p_r_e_s_s to compress news being transmitted, but
you don't want to.  _C_o_m_p_r_e_s_s is good and it is fast.

_L_i_b_r_a_r_i_e_s

     C News is mostly self-contained as far as libraries go.



                       3 Feb 1991





                           - 4 -


It  does need some things that are not yet universal, like a
full set of string functions (e.g. _s_t_r_t_o_k); we provide  rea-
sonably  portable  versions  of  these  for places that lack
them.

_N_e_t_w_o_r_k_i_n_g

     (We're talking here about networking in  the  sense  of
NFS and all those fun things, not _u_u_c_p or TCP/IP.) C News is
designed to run on systems which consist of a conglomeration
of machines on a local network, with only one of them acting
as news server.  Be warned, though,  that  if  you're  doing
this, you need to have _r_s_h or some reasonably equivalent way
of executing a program on another CPU.  If you've  only  got
one   machine,   just   make   sure   you   _d_o_n'_t   have   a
/_u_s_r/_l_i_b/_n_e_w_s/_s_e_r_v_e_r file and forget the whole issue.

_H_i_g_h_l_y _S_y_s_t_e_m-_S_p_e_c_i_f_i_c _T_h_i_n_g_s

     There are two small utilities within C  News  that  are
inevitably  highly system-specific and have a high probabil-
ity of needing changes to match your system.  Both  normally
live in /_u_s_r/_l_i_b/_n_e_w_s_b_i_n proper.

     _S_p_a_c_e_f_o_r is invoked by various components of C News  to
find  out  how much disk space is available.  The space mar-
gins in it are ones we find reasonable, but you may need  to
adjust  them.   On an old System V system in particular (one
where _d_f can't be applied to any directory name,  but  needs
to  be given a name in /_d_e_v), you will also probably need to
modify the locations to be _d_fed.  You will also need to  fix
_s_p_a_c_e_f_o_r  if your system's _d_f yields results in some strange
format or in some strange units.  We believe that we get  it
right for stock 4BSD, many (but probably not all) System Vs,
modern Irix, and real Unix (Version 7).  Beware  introducing
major  inefficiencies:  _s_p_a_c_e_f_o_r  is  called  a  lot.   (Our
current one could stand to  be  faster,  in  fact.)  Beware,
also, that _s_p_a_c_e_f_o_r is not intended to permit routine opera-
tion using filesystems that are on the brink of being  full;
the  limits  it imposes are only approximate, and no attempt
has been made to tune its  clients  to  exploit  every  last
available block.

     _Q_u_e_u_e_l_e_n is invoked by the  batcher  to  determine  how
long the current _u_u_c_p queues are, so it can judge whether it
should suspend batching of output to a given site.  There is
too much diversity in _u_u_c_ps for us to try to do this for all
possible versions.  We think we get it  right  for  the  two
most common flavors (HDB, aka BNU, and old subdirectory ver-
sions).  Our versions measure queue length in  batches,  not
bytes,  but  it  would not be hard to change this: _q_u_e_u_e_l_e_n,
the _b_a_t_c_h_p_a_r_m_s control file, and the  _s_e_n_d_b_a_t_c_h_e_s  how-many-
batches-should-I-try-to-add logic need to agree on the units
of measurement, but (we think) nothing else cares.



                       3 Feb 1991





                           - 5 -


     It is just possible  that  you  might  have  to  modify
_n_e_w_s_h_o_s_t_n_a_m_e,  which  also lives in /_u_s_r/_l_i_b/_n_e_w_s_b_i_n itself.
_N_e_w_s_h_o_s_t_n_a_m_e delivers the name which should  be  applied  to
the whole system (not just the particular CPU) for news pur-
poses.  It tries to be fairly clever about different ways of
getting  the  name,  and  in  particular will take it out of
/_u_s_r/_l_i_b/_n_e_w_s/_w_h_o_a_m_i if that  file  exists,  but  if  you're
doing  something  odd  on  a  strange system, changes may be
needed.

_I_n_p_u_t

     Input from the net via _u_u_c_p (or equivalent) shows up as
batches  of  news to be fed to _r_n_e_w_s or its obsolete synonym
_c_u_n_b_a_t_c_h.   These  two  programs  normally  live  in   /_b_i_n,
although  nothing  in the code knows about this and they can
go elsewhere (one of our systems does this).  Both are  sim-
ple  front  ends  that  invoke  _i_n_p_u_t/_n_e_w_s_s_p_o_o_l to store the
batch, while taking precautions to report trouble and not to
overflow  disks.   Neither  _r_n_e_w_s  nor  _c_u_n_b_a_t_c_h needs to be
setuid.

     Input via NNTP over the Internet (or  equivalent)  uses
rather  different  machinery  but  ends  up creating a saved
batch in much the same way as _i_n_p_u_t/_n_e_w_s_s_p_o_o_l does.

     _I_n_p_u_t/_n_e_w_s_s_p_o_o_l is a  small  C  program  that  saves  a
batch, writing into a file in /_u_s_r/_s_p_o_o_l/_n_e_w_s/_i_n._c_o_m_i_n_g.  It
must be able to create files there, and  _i_n_p_u_t/_n_e_w_s_r_u_n  (see
below)  needs to be able to read them and delete them.  This
gets a little tricky because _n_e_w_s_s_p_o_o_l will usually  be  run
by  _u_u_x_q_t  as  userid  _u_u_c_p (or something like that), not as
_n_e_w_s, which _n_e_w_s_r_u_n needs to run as.  The recommended  solu-
tion  is  to  have  _n_e_w_s_s_p_o_o_l  owned by _n_e_w_s and setuid.  An
alternative is to give the _i_n._c_o_m_i_n_g directory the userid of
_n_e_w_s and the groupid of _u_u_c_p, or vice versa, and set permis-
sions so that either can access it.  One of our systems  ran
that way for a while.

     To actually process incoming news,  _i_n_p_u_t/_n_e_w_s_r_u_n  gets
invoked  to  decompress the spooled batches and feed them to
_r_e_l_a_y/_r_e_l_a_y_n_e_w_s  (see  below).   There  is  an  option   for
_n_e_w_s_s_p_o_o_l  to  invoke _n_e_w_s_r_u_n when a batch is spooled, but a
(usually) preferable method is to have _c_r_o_n  invoke  _n_e_w_s_r_u_n
once  an hour.  _N_e_w_s_r_u_n does its own locking to prevent mul-
tiple  occurrences  running  simultaneously.   There  is   a
related  program, _i_n_p_u_t/_n_e_w_s_r_u_n_n_i_n_g, that can be used to set
or clear a flag that stops _n_e_w_s_r_u_n; this  may  be  a  useful
tactic  if  _n_e_w_s_r_u_n  should  not run at certain times.  Both
_n_e_w_s_r_u_n and _n_e_w_s_r_u_n_n_i_n_g must be run as _n_e_w_s.

     When a user posts news, he (or his news reader) does it
by feeding the article to /_b_i_n/_i_n_e_w_s.  In C News, _i_n_e_w_s is a
complex shell file that attends to  preliminaries  and  then



                       3 Feb 1991





                           - 6 -


invokes  _r_e_l_a_y/_r_e_l_a_y_n_e_w_s.   _I_n_e_w_s does not need to be setuid
(indeed, we make no use of setuid shell files at all,  since
they  are  grossly  insecure).   _R_e_l_a_y_n_e_w_s is the heart of C
News, the program that  actually  pulls  batches  apart  and
places  articles into the database.  If users are to be able
to post news, _r_e_l_a_y_n_e_w_s should be owned by _n_e_w_s and setuid.

_N_e_w_s _R_e_a_d_e_r_s

     C News is fully compatible with B  News  to  any  news-
reader  program  that  does  not inspect the middle field of
/_u_s_r/_l_i_b/_n_e_w_s/_h_i_s_t_o_r_y too closely.   Standard  B  News  news
readers  work fine.  We supply a simple news reader (written
by, and included with permission of, Michael  Rourke)  as  a
naive-user  replacement  for the B News _r_e_a_d_n_e_w_s.  More com-
plex programs are preferable for serious  news  enthusiasts.
We recommend Larry Wall's _r_n (which we use, unmodified), but
there are others.

_O_u_t_p_u_t

     _R_e_l_a_y/_r_e_l_a_y_n_e_w_s normally queues up news  for  transmis-
sion  to  other systems by appending article names and sizes
to     batch     files     in      subdirectories      under
/_u_s_r/_s_p_o_o_l/_n_e_w_s/_o_u_t._g_o_i_n_g.   These  are  then  processed  by
_b_a_t_c_h/_s_e_n_d_b_a_t_c_h_e_s, which should be run regularly,  as  _n_e_w_s,
by  _c_r_o_n.  _S_e_n_d_b_a_t_c_h_e_s can be configured to use a variety of
transmission mechanisms, the usual being _u_u_x.

_E_x_p_i_r_y _A_n_d _R_e_l_a_t_e_d

     News articles are removed, possibly with  archiving  to
an  archive  area, by the expiry subsystem.  _E_x_p_i_r_e/_d_o_e_x_p_i_r_e
should be invoked now and then, as _n_e_w_s, by _c_r_o_n.   We  sug-
gest nightly.  _D_o_e_x_p_i_r_e actually invokes _e_x_p_i_r_e/_e_x_p_i_r_e to do
the dirty work.

     C News _e_x_p_i_r_e does not have an option  to  rebuild  the
/_u_s_r/_l_i_b/_n_e_w_s/_h_i_s_t_o_r_y  file  from  scratch,  since  that has
nothing to do with expiry.  To rebuild _h_i_s_t_o_r_y, e.g.  if  it
has been destroyed, use _e_x_p_i_r_e/_m_k_h_i_s_t_o_r_y.

     Some news readers need  to  have  the  third  field  of
/_u_s_r/_l_i_b/_n_e_w_s/_a_c_t_i_v_e updated occasionally to show the lowest
article number still present in each newsgroup.  Frankly, we
think  such  news  readers  simply need to be fixed.  C News
_e_x_p_i_r_e does not do this updating.  For those  who  use  such
news  readers, however, _e_x_p_i_r_e/_u_p_a_c_t will do such an update.
It should be run as _n_e_w_s.  A much faster, but somewhat  less
portable, C implementation is supplied as _e_x_p_i_r_e/_u_p_d_a_t_e_m_i_n.

_R_e_b_o_o_t_s _a_n_d _A_d_m_i_n_i_s_t_r_a_t_i_o_n

     If the  system  crashes,  things  like  locks  must  be



                       3 Feb 1991





                           - 7 -


cleaned  out if C News is to function properly after reboot.
/_e_t_c/_r_c, or equivalent,  should  run  _m_a_i_n_t/_n_e_w_s_b_o_o_t  during
reboot, as _n_e_w_s.

     Certain log  files  can  grow  without  bounds  if  not
renamed/removed   now   and   then.   We  recommend  running
_m_a_i_n_t/_n_e_w_s_d_a_i_l_y once a day.  It tends  to  logs,  keeping  a
generation  or  two  for  use  in trouble tracking, and also
sends mail to the news administrator in the event that some-
thing funny has happened.

     In general, C News does not attempt to break locks,  on
the philosophy that a stale lock may mean something is badly
wrong.  (See _L_o_c_k_i_n_g  _i_n  _C  _N_e_w_s  for  details  on  locking
methods.)  The  various  programs  will  either  give up, to
return later, or wait patiently for the lock to go away.  If
one  doesn't  keep  an eye on things, a problem of some kind
can hang up the news system  for  quite  a  while.   Running
_m_a_i_n_t/_n_e_w_s_w_a_t_c_h  once  in a while-we recommend a few times a
day-will alert administrators to signs of  trouble.   Except
on  grossly  slow  systems,  C  News locks should never hang
around for any great length of time.



































                       3 Feb 1991


