This is a supplement to the main Readme file. The reader should consult
that file for more information.


Multiprocessor Architectures vs. Workstation Clusters
-----------------------------------------------------

	This software is based on (and compatible with) a patched 
version of PVM3.0 .  [3.2 - the two versions are now merged]

	In the workstation environment, there is usually one (or a few at
the most) task(s) and one PVM Daemon (pvmd) on each host. On a MPP
machine, however, only one pvmd runs on the front-end and it has to
support all the tasks running on the nodes. Node-to-node messages are 
sent directly across and very fast, but packets going to another 
machine must be relayed by pvmd and that link is slow, especially for
large messages.

	On a MPP machine tasks are always spawned on the nodes. If you
want to start a task on the front-end you have to run it as a Unix
process, which then connects to the pvmd by calling pvm_mytid().
However node processes not loaded by PVM are not allowed to
enrolled.

	The system supports on nodes of some MPP machines are limited.
For example if your program forks on a node, the behavior of the new
process is machine-dependent. In any event it would not be allowed
to become a new PVM task. 

	Inter-host task-to-task direct routing via a TCP socket connection 
has not been implemented on the nodes. Calling pvm_advise() on a node
has no effect, but turning on PvmRouteDirect in a host task could cause
problems.



iPSC/860
--------

	This port was developed on an iPSC/860 with 128 nodes, using an
SRM as its front-end. Compiling and installing on a remote host may
require extensive knowledge of Intel software.


INSTALLATION

	Type "make" on the SRM front-end to build pvmd and libpvm3.a, and 
		make PVM_ARCH=CUBE
to build libpvm3pe.a on a machine (SRM or workstation) that has the 
icc/if77 i860 cross-compilers.

	To make the examples, repeat the above steps in the "examples"
directory but use ../lib/aimk instead of make. (Note that aimk ---
and pvmd, pvmgetarch, etc. --- is a sh script. You have to run them
under sh instead of csh.)

	To build pvmd and libpvm3.a on a remote host (Sun/SGI) instead of
the SRM, try 
		make PVM_ARCH=I860
You may have to add a -L option to the definition of ARCHDLIB and a 
-I option to ARCHCFLAGS in conf/I860SUN4.def (or conf/I860SGI.def) 
to include the path of libhost.a and cube.h .



APPLICATION PROGRAMS

	Host (master) programs should be linked with pvm3/lib/I860/libpvm3.a 
and the SRM system library libsocket.a, node (slave) programs with 
pvm3/lib/ARCH/libpvm3pe.a and libnode.a .  Fortran programs should
also be linked with pvm3/lib/I860/libfpvm3.a (host program) or
pvm3/lib/I860/libfpvm3pe.a (node program).

	The programs in the gexamples directory have NOT been ported. The
user must modify the makefiles to link to the appropriate libraries.


BUGS AND TIPS

	1. On the SRM pvm is slow to start and it may sometimes time out and
print out the message "can't start pvmd", while in fact pvmd has already
been started. When this happens, just run pvm again and it'll connect
to the pvmd. If you start the pvm console on another machine and add the
I860 host to the configuration then this should not occur.

	2. To use PVM on a remote host you have to start "pvm3/lib/I860/pvmd"
manually in the background, because PVM has no way of knowing that your
workstation is being used as a front-end to the hypercube. PVM relies on
the shell script "pvm3/lib/pvmd" to determine the architecture of your
system, you can hack the script to make it do what you want.

	3. If PVM cannot obtain a cube of the size you specify because none
is available, it will return the error "Out of resources".

	4. The function pvm_spawn() will NOT return until all spawned tasks
check in with PVM. Therefore slave tasks should call pvm_mytid() before
starting any heavy duty computation.

	5. The only signal that can be sent to tasks running on the cube is
SIGKILL. Any other signals will be ignored.

	6. All spawned tasks should call pvm_exit() just before they exit.
Pvmd will release the cube after all tasks loaded into the cube have
called pvm_exit(). If any task attempts to continue execution after
it called pvm_exit(), it'll be terminated when the cube is released.

	7. There is a constant TIMEOUT in the file "pvmmimd.h" that controls 
the frequency at which the PVM daemon probes for packets from node tasks.
If you want it to respond more quickly you can reduce this value. 
Currently it is set to 10 millisecond.

	8. DO NOT use any native NX message passing calls in PVM or they
may interfere with pvm_send()/pvm_recv().

	9. Messages printed to "stderr" by node tasks do not end up in
the log file "/tmp/pvml.uid".

	10. The "argv" argument to pvm_spawn() is quietly ignored.



Paragon
-------

	This port has been tested on a Paragon XPS5 running R1.0c. The support
and information provided by Intel engineers are gratefully acknowledged.


INSTALLATION

	Type "make" in this directory on the Paragon. To build on a cross-
compiler machine, you need to type
		setenv PVM_ARCH PGON
		make -e
The environmental variable PARAGON_XDEV must be set to the proper base 
directory in the later case.

	To make the examples, go to the "examples" directory and repeat the
above procedure but use ../lib/aimk instead of make.


APPLICATION PROGRAMS

    Host (master) programs should be linked with pvm3/lib/PGON/libpvm3.a
and the system library librpc.a, node (slave) programs with
pvm3/lib/PGON/libpvm3pe.a and librpc.a .  Fortran programs should
also be linked with pvm3/lib/PGON/libfpvm3.a .

	The programs in the gexamples directory have NOT been ported. The
user must modify the makefiles to link to the appropriate libraries.


BUGS AND CAVEATS

	1. In T9 parallel applications cannot be started with 'rsh', so this
is not a "true" PVM. The PVM daemon cannot be started from other machines,
neither can it start pvm daemons on other machines.

	2. The pvm function "pvm_spawn()/pvmfspawn()" creates tasks on the 
nodes by default. The host (master) process must be run from Unix and 
then use the "pvm_mytid()/pvmfmytid()" function to enroll in PVM.

	3. In T9 parallel applications can hang for various reasons. If this
happens and no one else is using the machine, consult your Intel engineers.
If you cannot start the pvm console, try running pvmd with the trace 
function enable (i.e. pvmd -dff). You can also determine the state of 
pvmd with the Unix command "ps".

	4. By default PVM spawns tasks in your default partition. You can use
the NX command-line options such as "-pn partition_name" to force it to
run on a particular partition or "-sz number_of_nodes" to specify the
number of nodes you want it to use. Setting the environmental variable 
NX_DFLT_SIZE would have the same effect. For example starting pvmd with
the following command
		pvmd -pn pvm -sz 33
would force it to run on the partition "pvm" using only 33 nodes (there
must be at least that many nodes in the partition).

	6. The T9 operating system is rather fragile. To ensure your safety
try to run only one PVM master/slaves application at a time. DO NOT spawn
more tasks than the number of nodes allocated to PVM. If PVM appears hung,
halt it (or kill it if necessary) and restart. Be sure to test the examples
provided before attempting to run your own applications.

	7. There is a constant TIMEOUT in the file "pvmmimd.h" that controls 
the frequency at which the PVM daemon probes for packets from node tasks. 
If you want it to respond more quickly you can reduce this value. 
Currently it is set to 10 millisecond.

	8. DO NOT use any native NX message passing calls in PVM or they
may interfere with pvm_send()/pvm_recv().

	9. Messages printed to "stderr/stdout" by node tasks do not end up in
the log file "/tmp/pvml.uid".



CM5
---

	This port was developed on a 32-node CM5 running CMOST v7.2 and
CMMD3.0. We gratefully acknowledge the supports and information provided
by engineers of Thinking Machines Corp.


INSTALLATION

    Type "make" in this directory.

	To make the examples, go to the "examples" directory and type
		../lib/aimk default


APPLICATION PROGRAMS

    Host (master) programs should be linked with pvm3/lib/CM5/libpvm3.a .
Node (slave) programs need to be linked to pvm3/lib/CM5/libpvm3pe.a and 
joined with pvm3/lib/CM5/pvmhost.o and pvm3/lib/CM5/libpvm3.a using the
CMMD loader "cmmd-ld". Fortran programs should also be linked with 
pvm3/lib/CM5/libfpvm3.a .

    The programs in the gexamples directory have NOT been ported. The
user must modify the makefiles to link to the appropriate libraries.


BUGS

	1. The function pvm_kill() does not behave like a Unix kill: it
will delete the target task from PVM's task queue but doesn't actually
kill the task. 

	2. Signals sent by pvm_sendsig() are simply discarded, except for
SIGTERM and SIGKILL, which are equivalent to calling pvm_kill().

	3. You're not allowed to spawn more tasks than the number of nodes
available in your partition in one call to pvm_spawn().

	4. If you kill the node tasks from PVM (e.g., using pvm_kill), the
node processes will dump core files into your HOME directory. I don't
consider this to be a PVM bug.
