








             B News to C News Transition Guide


                       Geoff Collyer

                    Software Tool & Die
                       Brookline, MA


                          _A_B_S_T_R_A_C_T

          We describe the changes in  installation  and
     administration   from  B  News  to  C  News.   The
     intended audience is B News administrators switch-
     ing to C News.



_I_n_t_r_o_d_u_c_t_i_o_n

     C News was written by two  people  (Geoff  Collyer  and
Henry  Spencer)  who  were experienced B News administrators
(since 1982).  C News was intended to be a much  faster  and
more  correct  B News with some of the things that irritated
us in B News changed.  C News is thus administratively simi-
lar  to  B  News, though not identical.  We have no personal
experience running B News later than 2.10.1,  and  thus  may
have missed subtleties in later releases, notably 2.11.

     The single largest visible change from B News is  prob-
ably  that we wanted to allow the possibility of machines of
different architectures sharing  a  single  news  data  base
(/_u_s_r/_s_p_o_o_l/_n_e_w_s  plus  the data files in /_u_s_r/_l_i_b/_n_e_w_s in B
News terms) via network file systems while keeping  separate
directories  of executables elsewhere.  We did this by push-
ing the executables  into  a  new  directory,  often  called
/_u_s_r/_l_i_b/_n_e_w_s_b_i_n.  We also took the opportunity to push most
of the binaries into subdirectories to reduce the clutter at
the  top  level,  as the old B News /_u_s_r/_l_i_b/_n_e_w_s had become
awfully cluttered.

     The next most visible change is probably that  we  only
compile  in  default  values  for directory names (and a few
other configuration parameters), but they can be  overridden
at  run-time  via environment variables (subject to restric-
tions to prevent spoofing).   _n_o_t_e_b_o_o_k/_c_o_n_f_i_g  (relative  to
the C News source subtree root) describes the variables.  In
a nutshell, _N_E_W_S_C_T_L is the equivalent of /_u_s_r/_l_i_b/_n_e_w_s  when
referring  to  control  files,  _N_E_W_S_B_I_N is the equivalent of
/_u_s_r/_l_i_b/_n_e_w_s when referring to executables  (and  is  often
/_u_s_r/_l_i_b/_n_e_w_s_b_i_n),   and   _N_E_W_S_A_R_T_S  is  the  equivalent  of



                      January 10, 1992





                           - 2 -


/_u_s_r/_s_p_o_o_l/_n_e_w_s.  This capability can be used to run B  News
and  C  News in parallel for testing, feeding them identical
inputs.  It can probably be used instead of the _I_H_C_C  ifdefs
in  B News.  It is also used for regression testing, when we
run known input data through the  software  in  test  direc-
tories and verify that we get known results out; _m_a_k_e _r in a
given source directory will generally  invoke  the  relevant
regression test and _m_a_k_e _r_s in _c_o_n_f will invoke all of them.

_C_h_a_n_g_e_s _i_n _c_o_n_t_r_o_l _f_i_l_e _f_o_r_m_a_t_s

     In general, _n_e_w_s(5) now describes the file formats.

     _S_y_s adds a few optional features and withdraws some old
and  little-used ones.  Many obsolete flags draw fatal diag-
nostics, notably _N, the old,  unbatched  ihave/sendme  flag.
The other unsupported flags are _A (convert outgoing messages
to A News format; _r_e_l_a_y_n_e_w_s is not a protocol converter),  _H
(append  history  entry to named file?; sorry), _S (duplicate
the work of the shell to avoid one process; just delete this
flag),  _M, and _O (multicast flags; obsoleted by the batcher,
as far as we can tell).  New flags are _f (append spool  file
name  and  size  in  bytes  to the named file) and _n (append
spool file name and Message-ID to the named file; equivalent
to  _F_I  in  B  News).   There have been some changes to flag
semantics:  _I  in  B  News  invoked  all  the   ihave/sendme
machinery  _a_n_d  wrote  message-ids to the batch file; C News
just writes the message-ids.  See below for how  to  set  up
ihave/sendme feeds.  In C News, _F_I and _I are equivalent; use
_n if you want file names and message-ids.   All  file  names
written are relative to _N_E_W_S_A_R_T_S.

     B News allowed comments to be  continued;  this  is  at
odds with all other UNIX|- software, including  C  News.   To
comment-out  an  entry under C News, you need to prepend `#'
before _e_a_c_h line of the entry, not just the first.

     There is now an optional _d_i_s_t_r_i_b_u_t_i_o_n_s list  after  the
newgroups  pattern, separated by a slash.  If the fourth _s_y_s
field  is  a  relative   command   name,   _N_E_W_S_C_T_L/_b_i_n   and
_N_E_W_S_B_I_N/_r_e_l_a_y  will  be  searched before the standard search
path.   If  the  fourth  field  is  a  relative  file  name,
_N_E_W_S_A_R_T_S/_o_u_t._g_o_i_n_g  will  be  prepended.   The  ihave/sendme
kludges of B news have been expunged; one must say what  one
means, using three to five entries, typically.  See the sec-
tion below on ihave/sendme.

     _A_c_t_i_v_e has four fields  (there  is  vestigial  code  in
some, but not all, programs for two or three fields) and any
number of digits in the article-number fields (five  is  too
few  for  the long term).  Two new values are understood for
_________________________
|- UNIX is a trademark of Bell Laboratories.




                      January 10, 1992





                           - 3 -


the fourth (flags) field: _x, meaning ``quietly discard arti-
cles  for  this group'', and =realgroup meaning ``file arti-
cles for this group under _r_e_a_l_g_r_o_u_p instead''.  = is  useful
for  coping  with  badly-run local newsgroups, often created
from mailing lists.  You must  lock  the  news  system  with
_l_o_c_k_n_e_w_s before editing the active file.

     _H_i_s_t_o_r_y is in an  extended  B  2.10  news  format:  the
second field consists of two subfields separated by a tilde:
time received as an integer (a  _t_i_m_e__t  in  fact),  and  the
value  of  the _E_x_p_i_r_e_s: header, "-" if none.  More subfields
may appear in future.  You must lock the  news  system  with
_l_o_c_k_n_e_w_s  before  editing  the  history  file, and run _m_k_d_b_m
afterward to rebuild the index files.

     _L_o_g and _e_r_r_l_o_g are only faintly similar to their B News
counterparts.   Error  processing  in  general is due for an
overhaul,  which   may   further   revise   their   formats.
Currently,  _e_r_r_l_o_g  is  the  unadorned  standard  error from
_r_e_l_a_y_n_e_w_s(8) and any programs  it  runs.   _L_o_g  is  a  time-
stamped  log, down to the millisecond (where possible, offer
void where prohibited by System V), consisting of  one  line
per  article,  and  always showing the name of the host that
handed us this article and the Message-ID of  this  article.
See _n_o_t_e_b_o_o_k/_l_o_g for details.

     Locks are in the style of B 2.10: _N_E_W_S_C_T_L/_L_O_C_K  is  the
news  system  lock,  but  it never times out.  One locks the
news system by repeatedly attempting to  link  to  the  lock
name until successful; one releases the lock by removing the
lock name.  There are other locks, all of which  begin  with
the  prefix  _N_E_W_S_C_T_L/_L_O_C_K.  This scheme works well over net-
work file systems,  can  be  used  from  shell  scripts  and
interactively,  permits  one to see trivially what locks are
present,  and  is  portable  across  UNIX   variants.    See
_n_o_t_e_b_o_o_k/_n_e_w_s_l_o_c_k for details.

`_M_i_s_s_i_n_g' _c_o_n_t_r_o_l _f_i_l_e_s

     _A_l_i_a_s_e_s does not exist, since we  believe  that  header
munging  is  to  be avoided if at all possible (we do update
_P_a_t_h:, regenerate _X_r_e_f:,  and  delete  some  large  obsolete
headers  [_R_e_l_a_y-_V_e_r_s_i_o_n:,  _P_o_s_t_i_n_g-_V_e_r_s_i_o_n:, _D_a_t_e-_R_e_c_e_i_v_e_d:,
_R_e_c_e_i_v_e_d:, _P_o_s_t_e_d:, _I_l_l_e_g_a_l-_O_b_j_e_c_t:]).  It  is  possible  to
file  articles  locally under different newsgroups by use of
the _a_c_t_i_v_e file = flag.

     _N_o_t_i_f_y is replaced by _N_E_W_S_M_A_S_T_E_R in _N_E_W_S_C_T_L/_b_i_n/_c_o_n_f_i_g.

     _M_o_d_e_r_a_t_o_r_s, _r_e_c_m_a_i_l and _r_e_c_n_e_w_s are missing;  we  don't
know what they do (yet).

     _S_e_q is gone; we don't need it.




                      January 10, 1992





                           - 4 -


_N_e_w _c_o_n_t_r_o_l _f_i_l_e_s

     _B_a_t_c_h_p_a_r_m_s  controls  outbound  batching.   See   _n_e_w_s_-
_b_a_t_c_h(8).

     _E_x_p_l_i_s_t controls  expiry  policies.   The  command-line
options  for  _e_x_p_i_r_e  are  completely different from B News.
See _e_x_p_i_r_e(8).  The simplest possible _e_x_p_l_i_s_t is ``_a_l_l  _x  _7
-'' for 7-day expiry of all articles and history entries.

     _M_a_i_l_n_a_m_e is the domain name of this  machine  for  mail
purposes;  _w_h_o_a_m_i is the news name of this machine (e.g. for
_P_a_t_h: headers).   If  _r_e_p_l_y_u_s_e_p_a_t_h  exists,  automated  mail
replies  will use the _P_a_t_h: header for return addresses.  If
_s_e_r_v_e_r exists, _i_n_e_w_s will _r_s_h to the hostname found  therein
to run _r_e_l_a_y_n_e_w_s.

     _S_e_t_n_e_w_s_i_d_s is an optional setuid-root program  (in  the
wrong  directory,  it appears) to set user and group ids; it
should be obsolete now, but instead POSIX and System  V  are
making it necessary all too often.

_i_h_a_v_e/_s_e_n_d_m_e _f_e_e_d_s

     We didn't understand the implementation of ihave/sendme
in  B  News  for a long time and nearly didn't implement it,
having a very dim view of ihave/sendme to start with, but we
wanted  to  be  RFC  1036  compliant.   Upon discovering the
tricks (including mandatory default values for  various  _s_y_s
file   fields)  used  internally  by  B  News  to  implement
ihave/sendme, there was immediate agreement that

(a)  we should document how ihave/sendme works, and

(b)  we should implement something  more  obvious  and  that
     would  not  mandate  fixed values for fields that users
     might legitimately want to change  from  their  default
     values.

     We also see no use for the old  unbatched  ihave/sendme
protocol:  it  is  grossly  wasteful of resources at today's
traffic volumes and offers no  clear  advantage  to  sending
lists of Message-IDs in each _i_h_a_v_e or _s_e_n_d_m_e message.

     The  result  is  that  we  have   implemented   batched
ihave/sendme  as multiple sys file entries, which is clearer
but more verbose.  One typically needs an entry to specify

(1)  which articles are to be offered to  the  other  system
     via an _i_h_a_v_e message,

(2)  that _i_h_a_v_es and _s_e_n_d_m_es  are  to  be  batched  or  sent
     immediately,




                      January 10, 1992





                           - 5 -


(3)  that the response to an _i_h_a_v_e should be a  _s_e_n_d_m_e  mes-
     sage posted to the appropriate _t_o.site group, and

(4)  that the response to a _s_e_n_d_m_e message should be  batch-
     ing the requested articles.

     Note that any article requested by  a  _s_e_n_d_m_e  will  be
sent;  there is no checking that the requesting site is per-
mitted by its appropriate _s_y_s entry to receive this article.
Such  checking  could be added, at some cost in performance.
For now, disable  ihave/sendme  on  systems  with  `private'
newsgroups (if that isn't an oxymoron).

     See _n_o_t_e_b_o_o_k/_i_h_a_v_e for gory  details  including  sample
_s_y_s files.

_D_i_f_f_e_r_e_n_c_e_s _i_n _b_e_h_a_v_i_o_u_r

     The _c_h_e_c_k_g_r_o_u_p_s control message is non-destructive;  it
merely  mails  its output to _N_E_W_S_M_A_S_T_E_R (e.g.  _u_s_e_n_e_t).  The
distributed _c_h_e_c_k_g_r_o_u_p_s script has problems, which  will  be
fixed some day.  In the mean time, some people have reported
that the latest B News _c_h_e_c_k_g_r_o_u_p_s  script  works  fine  for
them; you could try it.

     There is an _A_l_s_o-_C_o_n_t_r_o_l: header recognised,  of  which
_S_u_p_e_r_s_e_d_e_s: is a special case.

_I_n_t_e_r_a_c_t_i_o_n_s _w_i_t_h _N_N_T_P

     We had erroneously assumed when writing C News that the
NNTP situation was not too disastrous.  Since we didn't then
run NNTP ourselves, we benignly ignored NNTP.  Only once  we
started  running  NNTP  did  we  realise  what a performance
disaster it was.  We are  still  working  on  improving  the
situation.

     The _C_N_E_W_S ifdef of NNTP as distributed by  Stan  Barber
makes  the situation tolerable (I wrote the original version
of the code inside the ifdef) if not wonderful.  In particu-
lar, it will happily accept duplicate articles, only to have
_r_e_l_a_y_n_e_w_s throw them away.  We have  an  experimental  _n_n_t_p_d
based  on  NNTP  1.5.8  that  we  are reasonably happy with,
though lock contention may still be  an  issue  for  a  busy
site.   Alternately,  Paul  Vixie's _m_s_g_i_d_d or some code from
Rich Salz may offer acceptable solutions.

     More work is called for.  Several  people  have  simul-
taneously  invented  the idea of a multi-threaded _n_n_t_p_d han-
dling multiple inbound transfer sessions and that may  be  a
worthwhile direction to pursue.  (It permits cheap duplicate
rejection with little lock contention and few processes.) It
has  the  disadvantage of not exploiting parallelism on mul-
tiprocessor machines, which seem to be what the biggest news



                      January 10, 1992





                           - 6 -


relay sites are becoming.

_d_b_m & _d_b_z

     There are now  several  redistributable  _d_b_m(3)  clones
available.   Before they appeared, we fixed and enhanced Jon
Zeeff's _d_b_z library; it is faster and has smaller files than
any  _d_b_m  or  _d_b_m  clone, so we recommend using it for news.
Whatever you elect to use, be sure  to  link  NNTP  and  any
other software that makes _h_i_s_t_o_r_y file lookups with the same
library that C News uses (picking up _l_i_b_c_n_e_w_s._a  is  usually
simplest).

_M_i_g_r_a_t_i_o_n _t_o _C _N_e_w_s

     A good first step is to read the documentation in the C
News  _d_o_c  source  directory.  You will want to refer to the
contents of the _n_o_t_e_b_o_o_k and _m_a_n directories.  If  you  have
trouble  formatting  any  of this documentation, get the _a_w_f
formatter and use it.  _A_w_f  can  be  had  from  your  nearby
_c_o_m_p._s_o_u_r_c_e_s._u_n_i_x archive site, or by _u_u_c_p or _f_t_p from _u_u_n_e_t
(_u_u_n_e_t._u_u._n_e_t)  as  _c_o_m_p._s_o_u_r_c_e_s._u_n_i_x/_v_o_l_u_m_e_2_3/_a_w_f._Z.    For
side-by-side comparisons of B and C News where this document
may have missed something, one could compare the correspond-
ing  sections  for B and C News in _M_a_n_a_g_i_n_g _u_u_c_p _a_n_d _U_s_e_n_e_t,
by Tim O'Reilly and Grace  Todino,  O'Reilly  &  Associates,
1989,  ISBN  0-937175-48-X,  inquiries  to `_n_u_t_s@_o_r_a._c_o_m' or
`_u_u_n_e_t!_o_r_a!_n_u_t_s', or 800-338-NUTS (in  California,  800-533-
NUTS),  FAX  707-829-0104.   See  _d_o_c/_b_i_b_l_i_o  for  a  fuller
bibliography.

     It is probably safest to create parallel C News  trees,
at  least for the equivalent of /_u_s_r/_l_i_b/_n_e_w_s, then populate
them and rename the B and C trees  when  you  are  satisfied
that  C  News is installed and set up correctly.  The C News
installation procedure (_b_u_i_l_d) will not  overwrite  existing
control  files  in _N_E_W_S_C_T_L, so copying your old _a_c_t_i_v_e, _s_y_s,
and possibly _h_i_s_t_o_r_y  files  to  _N_E_W_S_C_T_L  adapting  them  as
necessary (see below), and then installing C News there nor-
mally should get you started.

     One advantage of setting up a parallel tree for C  News
is  that  it  is possible, though _b_u_i_l_d doesn't know how, to
run all incoming news into both news systems for  a  day  or
two  to  convince yourself that everything is working right.
If you decide _n_o_t to switch, you can just remove the C  News
tree.   In  any  case,  running  both systems simultaneously
avoids the desperate panic of having to cut everything  over
at  once  and  get it right first time to avoid losing news;
this just should not be necessary any more.  The  change  is
just  to  ensure  that  all the commands that used to invoke
_r_n_e_w_s (_i_n_e_w_s, _r_n_e_w_s, _c_u_n_b_a_t_c_h and probably a few others) now
hand  the  incoming  batch  to both news systems.  Something
like this should suffice (and has worked in the past):



                      January 10, 1992





                           - 7 -



        # parallel rnews
        f=/tmp/rn$$
        cat >$f
        brnews <$f
        crnews <$f
        rm $f


     _S_y_s should only need  to  be  scanned  for  unsupported
flags  (I  think  _S can just be deleted without ill effect),
and entries with _I flags rewritten as multiple  ihave/sendme
entries.   _A_c_t_i_v_e  can optionally be edited to refile groups
locally, to compensate for lack of _a_l_i_a_s_e_s.   C  _e_x_p_i_r_e  can
read a correct B _h_i_s_t_o_r_y file, but B News sometimes produced
incorrectly-formatted history files.  To  be  safe,  and  to
pick  up information that B News didn't store in the history
file, run _m_k_h_i_s_t_o_r_y to  regenerate  _h_i_s_t_o_r_y  and  its  index
files.   _M_k_h_i_s_t_o_r_y  will  take quite a while; it has to open
every article in the news system, which can  take  hours  on
some machines.

     The  normal  installation  procedure  is  described  in
_d_o_c/_i_n_s_t_a_l_l  and  really  just  involves running _c_o_n_f/_b_u_i_l_d,
answering its (many) questions, and following  its  instruc-
tions.   (We  intend  that eventually _b_u_i_l_d will be run once
per site for all time (or until the hardware  or  OS  change
drastically)  and  that  thereafter  components may be built
using _m_a_k_e.) Do be careful to use  the  correct  userid  for
each  step  of  installation;  doing the whole thing as _r_o_o_t
will result in incorrect file  ownerships.   Two  major  by-
products  of  _b_u_i_l_d are _l_i_b_c_n_e_w_s._a in the root of the source
subtree and the directory _i_n_c_l_u_d_e.

     There are some  differences  in  configurable  options.
All  the  length  and size limits are gone.  Essentially all
the B News _m_a_k_e_f_i_l_e options are now queried  for  by  _b_u_i_l_d.
DFLTEXP  and  HISTEXP  are now specified in _e_x_p_l_i_s_t.  Only a
trivial newsreader, a _r_e_a_d_n_e_w_s replacement, is included with
C  News,  so little of the reader configuration is relevant.
NOTIFY is now _N_E_W_S_M_A_S_T_E_R and _b_u_i_l_d queries for it.   DFTXMIT
and  UXMIT  are now CMDPFX and CMDSFX in _r_e_l_a_y/_s_y_s._c with no
simple override (though an override is in  the  works);  the
default  command,  in case anyone still uses it, is _u_u_x - -_r
-_z  system!_r_n_e_w_s.   MANUALLY,  NONEWGROUPS  and  UUPROG  are
replaced  by  policy  in  shell scripts; edit _r_e_l_a_y/_c_t_l/* to
taste.  BATCH is gone; the unbatcher is  built  into  _r_e_l_a_y_-
_n_e_w_s.   OLD is gone (_r_e_l_a_y_n_e_w_s is not a protocol converter).
DOXREFS is gone; _X_r_e_f: is always generated for  cross-posted
articles.   SENDMAIL  and  MMDF  are gone; we just use _m_a_i_l.
DEADTIME is gone; locks do not time out.

     This table summarises the disposition of the  remaining
B  News  options.  File names in the `build file' column are



                      January 10, 1992





                           - 8 -


relative to _N_E_W_S_C_T_L and refer to files generated  by  _b_u_i_l_d.
A `flag' in the `note' column indicates that the presence or
absence of the file _i_s the flag.  A  `gone'  in  the  `note'
column  indicates  that  the  option  has  vanished.  If the
`build file' column is  empty,  the  option  has  just  been
absorbed into _b_u_i_l_d.

box,  center;  c  c  c  l  l  c  .   B   option        build
file      note     _    SPOOLNEWS       rnews.newsrun   flag
INTERNET        replyusepath    flag
MYDOMAIN        mailname MYORG   organization UUNAME  whoami
_     CHEAP           gone      NICENESS                gone
FASCIST         gone            ORGDISTRIB              gone
MULTICAST               gone _ UNAME GHNAME  BSD4_2  BSD4_1C
LOCKF HIDDENNET SMALL_ADDRESS_SPACE


     If you compile and install the software as  two  steps,
it is simple, though time-consuming, to run regression tests
for the major subsystems: _c_d into a source  directory  (e.g.
_e_x_p_i_r_e,  _b_a_t_c_h,  _r_e_l_a_y) and type _m_a_k_e _r.  If you get no com-
plaints, the software is believed (after  testing)  to  work
correctly on your machine.

     Sites with a lot of outgoing news feeds (over 60, say),
should  be  aware that batch file writing will probably dom-
inate _r_e_l_a_y_n_e_w_s's elapsed time.  We hope to fix this.

     After installation, errors will generally  be  reported
by  mail  (to  _N_E_W_S_M_A_S_T_E_R [typically _u_s_e_n_e_t]).  To watch the
progress   of   incoming   news,   tail   _N_E_W_S_C_T_L/_l_o_g    and
_N_E_W_S_C_T_L/_e_r_r_l_o_g.   _e_r_r_l_o_g  should  be  empty  on  a smoothly-
running system.
























                      January 10, 1992


