$Id: readme 1.12 1996/10/06 19:30:19 hardy Exp $	

THIS PROGRAM IS HEAVILY BASED ON IDEAS/MODULES/DOCS FROM CHIN HUANGs
(cthuang@io.org) SOUPER15.  THANKS TO HIM.  ALSO FOR YARN/2.

Nevertheless, all bug reports and everything else concerning VSoup
should be submitted to Hardy Griech (rgriech@ibm.net), not to Chin
Huang!

VSoup is free software.  See COPYING disclaimer of the FSF for
details.  Note that you are not allowed to distribute VSoup without
source code.  Also there is no warranty for correct functioning of
VSoup.



Purpose
=======

VSoup is a network mail and news client program for OS/2 Warp's
Internet Access Kit.  It converts mail and news fetched from a
POP3/NNTP server to SOUP format.  It can also send mail/news messages
contained in SOUP reply packets to NNTP/SMTP servers

Currently there are no man pages for VSoup.  The man pages from
souper15, the usage message and this README should be enough.

--------------------------------------------------------------------------------
The program requires the emx run-time DLLs.  You can get these from

     ftp://ftp-os2.nmsu.edu/os2/unix/emx09c/emxrt.zip

The emx09c run-times or newer are required.  EmxRev should show at
least revision 50 for the DLLs.  VSoup works possibly with older
versions, but keep in mind, that there were many changes in the DLLs
concerning multi-threading / sockets since emx09a.
--------------------------------------------------------------------------------
!!!!!!!!!!!  TAKE CARE, THAT YOU ARE USING THE CORRECT DLL-VERSIONS  !!!!!!!!!!!
--------------------------------------------------------------------------------



Features
========

-  receiving news from NNTP, posting news to NNTP, receiving mail from
   POP3, sending mail to SMTP

-  all the packets are exchanged in SOUP format files

-  multi-threaded news receiving, several strategies for news
   receiving

-  ability to filter articles by header lines before retrieving the
   body text

-  status mail generation.  The generation can be forced via command
   line option, standard is to generate mail in case of failure or
   special events (e.g. cannot select newsgroup, new newsgroups, ...)

-  shows some statistic information about data transfer

-  detects too low throughput or stuck connections

-  URLs on command line for easy specification of the several servers

-  knows about NNTP-authentication

-  YarnIo.Cmd script to do VSoup operation safely (without risc of
   message loss etc.) and concurrently (EMail/News transmission /
   reception at the same time, News reception from two servers).  Give
   it a try!

-  FAQ included (in this README)

-  source code included

-  freeware according to GPL (without any warranty for correct
   function etc)



Warning
=======

Some news sites might see VSoup as a news sucking utility and might
feel offended by its use.  The responsibilty to use VSoup in a fair
way to other news users is in your hand - especially the other legal
VSoup users don't want a general ban of VSoup.  I explicitly forbid
the usage of VSoup in a sucking way, i.e. you are allowed to receive
news for your personal use from a responsible number of news groups.
You are NOT allowed to use VSoup for acquiring news for further
distribution purposes from your ISP without the written permission -
i.e. knowledge - from your ISP.



Installation
============

The installation description is really incomplete.  If anybody want's
to add something, please email.

(!)  get the latest version of the emx runtime lib and install it or
     check your current installation with 'emxrev'.  Revisions should
     show higher or equal than '50'.  If you are not intending to use
     YarnIo.Cmd, then add now 'SET EMXOPT=-h40' to your CONFIG.SYS and
     reboot!

(1)  unzip VSoup* in a temporary file directory (%TMP% is a good
     place)

(2)  move VSOUP.EXE (VSoup-executable), YARNIO.CMD (script for doing
     Yarn/VSoup-IO), YARNIO.SET (configuration file for YARNIO.CMD)
     into a subdirectory of your choice (using a subdirectory
     contained in the %PATH% is always a good choice).

     Note:  YARNIO.CMD & YARNIO.SET must reside in the same
            subdirectory and must have the same filename, except the
            extension...

     If you are interested in the source code, move it also to a
     specific subdirectory.

(3)  review the 'command line options'-section below and edit the
     YARNIO.SET configuration file according to your specific needs.
     Also read the comments in YARNIO.SET.

(4)  edit the other variables in the YARNIO.SET file.  If you like to
     let YARNIO.CMD dial for you, you must also provide the scripts
     LOGINISP.CMD and LOGOUTISP.CMD.  Perhaps also some other
     statement must be adapted.

     Note1:  the setup of YARNIO.SET is required only once.  Future
             updates will only belong to YARNIO.CMD

     Note2:  if you are not interested in a service
             (e.g. soupRcvNews2), set the corresponding variable to ''

(5)  setup the directory structure for YARNIO.CMD:
     - %HOME%\yarn\in       - files for reception
     - %HOME%\yarn\in\mail  - files for received mail
     - %HOME%\yarn\in\news  - files for received news
     - %HOME%\yarn\in\news2 - files for received news from alternative source
     - %HOME%\yarn\out      - souper packets for transmission

     The news & news2 subdirs contain the newsrc-files (the groups you
     have selected for reception).

(6)  change in the yarn-config file (%HOME%\yarn\config) the following line:

     reply-packet=%HOME%\yarn\out\reply.zip

     If this misses, reply will not be possible!  Of course %HOME%
     should be replaced by the actual value of %HOME%

(7)  check your %ETC%\SERVICES file, if it is containing the following
     lines:

     smtp              25/tcp mail      #Simple Mail Transfer
     smtp              25/udp mail      #Simple Mail Transfer
     pop3             110/tcp           #Post Office Protocol - Version 3
     pop3             110/udp           #Post Office Protocol - Version 3
     nntp             119/tcp readnews untp #Network News Transfer Protocol
     nntp             119/udp readnews untp #Network News Transfer Protocol

     If one (or more) is missing, you have to add the missing item(s).

(8)  start YarnIo

If you don't like YARNIO.CMD, simply skip steps (4)/(5)/(6)/(8) and
parts of (2)/(3).

Debugging Issues
----------------

-  generate the VSoup status mail with '-M' and check what it says

-  is the configuration file YARNIO.SET correct?

-  is the subdirectory setup correct?

-  is VSoup called in each window?

-  does the pingHost work as expected?  Issue the program via REXXTRY
   to check the return values in connected/disconnected case.

-  are LOGINISP.CMD / LOGOUTISP.CMD working?  Call them directly from
   a command line and check what happens



Command line options
====================

-h<dir>     set the home directory for the VSoup operation.  You can
            also specify your home directory through the environment
            variable HOME

-i          ignore the settings of the 'Internet Connection for OS/2'
            settings

-m          do not fetch mail

-M          generate the VSoup status mail in any case.  Standard is
            to generate the message in case of error only.

-n          do not fetch news

-r          read only mode.  Except the .\areas, .\*.msg, .\*.idx no
            files are written (e.g. %HOME%\newstime, %HOME%\newsrc).
            Also the mails are not deleted from the POP3 server.  Main
            purpose for this option is debugging.

-s          send replies.  -s might be combined with the -m,-n option,
            but this feature is ignored!

-T<n>       throughput must be higher than 'n', otherwise transfer
            will be aborted.  Throughput is checked against the limit
            'n' every 10s after data transfer has begun.  If 6 samples
            show not enough throughput, VSoup is aborted.  Default is
            '0 bytes/s', i.e. stuck connection.  Use '-T-n' to check
            your throughput conditions.


News receiving options
----------------------

-a          add new newsgroups to newsrc file.  If there are any new
            groups on the server, the generation of the status message
            will be forced.

-c[n]       catchup: mark every article as read except for the last n
            (default is 10).

-k<n>       set maximum news packet size to n KBytes (default is no
            limit).  N means the total amount of received messages,
            not the size of a single message (see -l)

-K<file>    set the name of the kill file (see note below...)

-l<n>       set the maximum length of news you like to receive.  This
            is intended for omitting binaries in text groups.

-N<file>    set the name of the newsrc file (see note below...)

-S<n>       receiving strategy (n=0..2).  Speed increases (at least
            should) in the direction 0 to 2, danger of receiving
            crossposted articles increases also from 0 to 2.  The
            default of 1 is ok for most cases, but this parameter is
            an issue of finetuning!

            Recommendation:

            -  If you have a lot of groups with small amount of
               articles per fetch and only a little bit cross-posting,
               '-S2' will be the best option for you.  If you allow
               reception of crossposted articles ('-x'), '-S2' will be
               best for you in any case.

            -  if you are accessing groups with large articles
               (i.e. cost of double reception of crossposted articles
               is high), '-S0' will be the best for you.

-t<n>       receive the news with 'n' threads, i.e. connect to the NNTP
            server 'n' times

-u          create news summary.  Output can be found in the .\*.idx
            files.  Be aware that the summarized articles are marked
            as read (one way to get out of this, is to call VSoup with
            -r)

-x          do not process news Xref headers, i.e. allow reception of
            crossposted articles


-K,-N,-h
--------

I do not recommend to use the options -K & -N, because behavior is not
very transparent if you are using '-h' too.  If the '-h' is behind the
'-K/-N' the result will be another as expected - VSoup (like souper)
will revert to 'kill' and 'newsrc'; if '-K/-N' are behind the '-h' and
do not contain absolute pathes, the files will not be located in the
by '-h' specified home directory.  This is confusing and will be fixed
in a future release.  IT IS BETTER AND CLEANER TO USE -h AND SETUP THE
SPECIAL KILL/NEWSRC FILES IN THAT DIRECTORY!


URLs
----

The addresses of the hosts are given in URL form.  These URLs confirms
not absolutely to the standard, so a short explanation follows:

-  'nntp://[UserId[:Password]@]nntp-host' defines the address of your
   NNTP server.  UserId and Password can be given if required for
   AUTHINFO.

-  'pop3://[UserId[:Password]@]pop3-host' defines the address of your
   POP3 server.  UserId and Password are usually required.

-  'smtp://[userid[:password]@]smtp-gateway' defines your SMTP
   gateway.  UserId and Password are not required, in fact they are
   ignored by VSoup!

And please: don't mix up news and EMail (your EMail might be news, but
in these terms they remain EMails).  EMail can be compared to the
postal service, just like a letter.  EMail-reception is handled
through POP3-server (pop3:// prefix), transmission through
SMTP-gateways (smtp:// prefix).  News are more like broadcasting,
everybody will hear your message.  News are transmitted and received
via NNTP-servers (nntp:// prefix).

Perhaps it is a little bit confusing, that you can also deliver NEWS
via EMail service (just like the yarn list).  The other way around is
not possible...

The URL-feature makes VSoup different to souper16, which wants to see
a %HOME%\newsauth file for NNTP-authorization.

Ah ja: it is not intended to implement other protocols like http or
so...


Evaluation Order Of URLs
========================

Evaluation order is as follows:

(1)  take all information from the 'Internet Connection for OS/2'
     settings

(2)  check the NNTPSERVER environment settings (yes, there are no vars
     for the SMTP/POP3 setup).  NNTPSERVER is a URL without leading
     'nntp://'

(3)  command line options

If you specify only e.g. 'nntp://news-server' in the command line
(without userid/password), the userid/password from the OS/2 settings
are taken as a standard value.



News Reception
==============

for news reception you have the following options (which cannot
combined): either process .\commands (.\commands exists), or catch-up
the newsgroups (-c option) or create newsgroups summary (-u option) or
receive the news in the standard way (no specific option).  New
newsgroups are added only to the %HOME%\newsrc file, if news received
in the standard way.



Environment
===========

ETC          ETC must be set for accessing the 'Internet Connection
             for OS/2' settings.  If '-i' is specified in the command
             line, ETC is not required.

HOME         your home directory (can also be specified on the command
             line).  If HOME is not defined, it defaults to '.'.

MAILER       command for transmitting EMails (the EMail is piped into
             the stdin of MAILER).  If VSoup should do the job, MAILER
             must be undefined.

NNTPSERVER   hostname of the nntp server (can also be specified in the
             command line).  This variable is to my opinion obsolete
             and remains only due to compatibility reasons

POSTER       command for transmitting news (the article is piped into
             the stdin of POSTER).  If VSoup should do the job, POSTER
             must be undefined.



Diagnostics
===========

VSoup returns with an exit code of '1', if any operation failed.
Otherwise return code will be '0'.

Failing operations are (not complete):
- could not setup connection (many reasons)
- could not deliver email/news
- timeout detected (connection broken)
- user hit ^C or ^BREAK
- ill options, ill setup (e.g. no ETC set, but 'Internet Connection'
  settings requested), ill OS (e.g. D*@!S)
- there was no areas file and you have requested '-s'

Failing operations are NOT (again not complete):
- there was no EMail in your POP3-mailbox
- there were no new news
- a newsgroup activated in newsrc does not exists (check StsMail.Msg)
- you have requested '-s', but a message file is missing (VSoup deletes
  message files immediately after successful delivery.  If you need
  two runs to deliver all your messages e.g. one file will be missing on
  the second attempt.  This is considered ok)

It is possible, that diagnostics code will change in the future.
Return codes with a cleared bit 0 (like 0,2,4,...) will indicate a
non-fatal condition, return codes with a set bit 0 will indicate fatal
conditions.



YarnIo.Cmd
==========

There is a REXX script packed into this archive.  The script is used
for IO of VSoup to YARN.  News are read from two NNTP servers,
mail/news are transmitted, mail is received - all simultaneously.
Actions can be disabled individually (type 'YarnIo ?'  for more
information).

The script should be started by the dialer directly after connection
has been established.  It is also possible to let it dial itself.
Therefor the two script 'loginisp.cmd' and 'logoutisp.cmd' must be
provided (you can change the names in the configuration file).

Most items of the user configuration are in a file called YarnIo.Set.
It contains the settings for the initiation of the external programs
including 'vsoup', 'import'... calls.  You have to adapt this file to
your needs for sure.

Sorry, but the directory structure must be adapted manually!  You
should follow the structure given in YarnIo.Cmd.



Files
=====

%HOME%\newsrc:   contains the newsgroups.  ':' behind the name
                 indicates subscribed group, '!' indicates an
                 unsubscribed one.  Syntax is as follows:

                 %HOME%\newsrc ::= <lines>

                 <lines> ::= <line> <lines> | empty

                 <line> ::= <groupname> <sep> <article-ranges> "\n"

                 <sep> ::= "!" | ":"

                 <article-ranges> ::= <article-ranges> "," <art-range> |
                                      <art-range>

                 <art-range> ::= " " <art-num> | " " <art-num> "-" <art-num>

                 <article-ranges> must be sorted in ascending order to
                 be recognized correctly.  Blanks are only allowed at
                 the indicated positions.

                 Under normal circumstances, only the <sep> is
                 important to the user for subscribing & unsubscribing
                 groups.

.\commands:      contains lines with the syntax: 'sendme' <groupname>
                 <article-nums>.  .\commands is executed instead of
                 standard fetching of news.  After successful
                 'commands'-processing, the file will be deleted.
                 Note: already read articles (marked in %HOME%\newsrc)
                 will be skipped!

%HOME%\kill:     is the kill file.  Syntax of the kill file is as
                 follows (not too pedantic):

 		 - %HOME%\kill ::= {<line> "\n"}*

                 - <line> ::= '#' <text>  | <kill-section>

                   lines beginning with a '#' indicates a comment
                             
                 - <kill-section> ::= "all"        "{" <kill-exps> "}" | 
                                      <group-name> "{" <kill-exps> "}"

                   "all" indicates, that the following <kill-exps> is
                   valid for all newsgroups, <group-name> limits the
                   <kill-exps> to a specific group.

                 - <kill-exps> ::= <header-name> <kill-exp> "\n"
                                   {kill-exps}

                   if <header-name> is "header", the <kill-exp> is
                   valid for the complete header.  <kill-exp> is a
                   regular expression.

                 Note1:  "\n" is the newline character.  It is
                         mandatory!

                 Note2:  Upper/lower case is ignored.

		 Note3:  News transmission speed decreases if a kill
                         file is used, because an article is fetched
                         then in two steps: HEAD <num>, then BODY
                         <num>.  Otherwise an article will be fetched
                         in only a single step: ARTICLE <num>.
                         Scoring from Yarn is also more flexible than
                         this simple killing, and anyway, who knows in
                         advance about the development of a thread...

%HOME%\newstime: contains the time of the last NNTP connection.  This
                 is for fetching of new newsgroups.  If
                 %HOME%\newstime does not exist, ALL currently
                 available newsgroups are fetched from the server.
                 Applicable only, if VSoup was called with the '-a'
                 parameter.

.\00*.msg:       contain the received news messages (news & mail)

.\00*.idx:       contain the received newsgroup summaries ('-u' option)

.\stsmail.msg:   contains the status message generated by VSoup

.\areas:         contains the table of contents of the received
                 .\00*.msg / .\00*.idx files (+stsmail.msg)

.\news.msg:      contains the news messages that should be sent to the
                 NNTP server

.\mail.msg:      contains the mail messages that should be sent to the
                 SMTP server

.\replies:       contains the table of contents of the news.msg /
                 mail.msg files to be transmitted



Insufficiencies
===============

-  only very basic killfile handling

-  not compatible with souper16 (some cmd options, some files,
   score/kill file)

-  the port section in URLs is not recognized (future version)

-  space allocated from the heap is not freed (debugging...)

-  behaviour for news receiving/posting, mail receiving/posting is not
   symmetrical, i.e. there is no multi-threading except for news
   receiving...  This will be fixed in a future version

-  only one single version (for OS/2)



Known Bugs
==========

-  how to kill a thread (I think, this is an old OS/2 issue)? (problem
   arises, if the main thread of VSoup has received a signal)

-  version of the emx runtime is not checked!

-  some write protected output files will kill VSoup (newsrc,newstime,?)



Plans
=====

-  remove bugs & insufficiencies

-  more situation dependent return codes 

-  use also the message-IDs for retrieving articles (this requires
   NEWNEWS on server side and also access to news.dat)



Author
======

Hardy Griech
Kurt-Schumacher-Str. 25/1
72762 Reutlingen
Germany

email:  rgriech@ibm.net



Acknowledgements
================
FSF               for GCC (2.7.2.1 used), EMACS (19.31.1 used)
Eberhard Mattes   for the GCC,EMACS ports to OS/2
Ralph Bednarsky   VSoup1.2 gamma tester



History
=======

VERSION 1.2
-----------

-  first non-beta release

-  more detailed README

-  YarnIo enhanced (autodial, disabling specific actions,
   configuration file, ...)

-  some changes/enhancements/extensions in StsMail.Msg, e.g. command
   line parameters are contain, important server names too... (useful
   for debugging)

-  if supported by the NNTP server, the XHDRs are retrieved to
   determine holes in the article sequence

-  throughput check (-T option) now replaces the old timeout
   detection.  It is now working for all operations.

-  news reading, strategy parameter:

   -S0  groups are read consequent sequentially, i.e. all threads are
        fetching articles from only one group

   -S1  the next group will be accessed, if one thread goes into
        waiting state - only small modification of the above

   -S2  it is tried, to keep all threads busy.  I.e. each thread
        accesses one dedicated group, groups are read parallel (if
        there are no more groups to access, the waiting threads are
        used for parallel reading from one group)

-  more robust (but slower) reading of %HOME%\kill and %HOME%\newsrc.
   Because it is slower: clean-up your newsrc from time to time, if
   you are using the '-a' option

-  bug fixes:

   -  'internet connection' settings were never ignored ('-i' did not
      work)

   -  some very minor bug fixes for handling of broken connections

   -  under some circumstances broken SMTP connection did not produce
      an error condition

   -  bug in kill file handling fixed (trailing blanks in line were
      not accepted).  Lines in the kill file can now be outcommented
      through a '#' as the first non-blank character

   -  if getArticle() fails, the according connection will be marked
      as failed.  This lets VSoup die gracefully, if single
      connections are cancelled (by whom?)

   -  signal handling changed: VSoup tries (!) to kill the sub-threads

   -  number of file handles provided by EMX is now checked

   -  now compiled with emx09c.  During this 'port' it became obvious,
      that streams are not appropriate.  Thus all file i/o is done
      directly via handles (class TFile in mts.cc)


VERSION 1.1 (010996)
---------------------

-  first public beta

-  if NNTP server knows nothing about DATE command, the internal clock
   will be taken as a reference (required for -a cmd option only)

-  'AUTHINFO USER' / 'AUTHINFO PASS' for nntp server implemented
   (RFC977 extension).  Call VSoup simply with the nntp-URL
   nntp://user:passwd@nntpserver

-  NNTP NEXT will only be done, if there is a bigger gap between the
   articles

-  bug in socket.cc ((char)0xff == (int)-1 !!!)


VERSION 1.0 (010896)
---------------------

-  Program is now named VSoup.  I am sorry, but the program again
   requires the emx DLLs (to my opinion no disadvantage, because most
   of the people will have them anyway).  Also there is no support for
   Win95 (this was not intended and I am not sorry, but I have no
   Win95 system - and I am happy about that ;-)

-  Program is now multi-threaded for news reading.  This gives a speed
   gain of 200%-500% depending on the overall speed of the connection
   and the number of threads; on the other hand multiple connections
   have a communication overhead (i.e. the connections must be
   established, the groups selected).  Estimated loss is around 5%...

-  Return codes are now much more consistent: on failure an
   EXIT_FAILURE (1) is returned, on success EXIT_SUCCESS (0)

-  On failure a status mail is generated.  Most of the console
   messages are redirected into this status mail.  The generation of
   the mail can be forced thru cmd option '-M'

-  If file %HOME%\NEWSTIME does not exist, the complete newsgroup list
   is retrieved from the news server (NNTP LIST)

-  Kill files may have comments ('#' in the first position of a line).
   This is very beta...

-  Readonly mode is now much more consistent (NEWSTIME is not updated,
   also the sent articles are not deleted)

-  the default maximum packet size has been changed to unlimited

-  Small bug in newsrc handling found (firstUnread was wrong
   sometimes)

-  automatic timeout for NNTP reception



Small FAQ
=========

Q1: Why do I get something like '0000002.msg: Too many open files'?
Q2: I am getting the message '...number of threads cut...'.  What does
    it mean?

A:  VSoup needs a lot of open file handles.  Thus it is recommended to
    add the following line to your CONFIG.SYS: 'SET EMXOPT=-h40 -c
    -n'.  This allows 40 open file handles which should be fairly
    enough.  YarnIo is doing that automatically.


Q:  Why is my news reception getting stuck sometimes?

A:  One common reason is that you are using the wrong (too old)
    version of the emx-runtime-DLLs.  Check them with 'emxrev' (should
    show revisions >= 50) and update them if required.


Q:  I cannot receive EMail from my pop3-server?

A:  - Check, if your UserId/Password settings are ok
    - Check, if your %ETC%\services file contains the following lines
      (see also installation):

      pop3             110/tcp           #Post Office Protocol - Version 3
      pop3             110/udp           #Post Office Protocol - Version 3

      Yes, this is a small incompatibility to Souper...


Q:  Give me an example killfile!?

A:  # this is a killfile for VSoup
    #
    all {
       header microsoft
       from bill.*gates
       subject make.*money
       subject \$\$\$
       x-newsreader forte agent
    }

    comp.os.os2.mail-news {
       subject ultimail
       subject netscape
       subject help:
    }


Q:  I have 8 connections requested for news receiving (with the -t8
    option), but the status message says that only 3 threads were
    connected successfully.  What's wrong?

A:  Nothing!  Sometimes it takes very long to establish a connection
    to the news server which means that VSoup has finished before all
    requested threads are connected.  Sometimes (I don't have heard of
    that case, but who knows) the news server only grants e.g. 3
    simultaneous connections by one user.


Q:  How do I get a listing of all available newsgroups?

A:  Delete the file %HOME%\newstime.  After next connection you will
    get a status mail and the new/available groups are appended to
    your newsrc file.
