\ ( Networks of Neurons     Robert E. La Quey   FORML 87 )

\ The following is source code presented by Robert La Quey
\ at the FORML conference in November, 1987.  This was
\ released into the public domain so that people could become
\ familar with neural nets and how they could be done.

\ I've written it up in block sections groupings as it was 
\ presented in the paper.  The shadow blocks are included at the 
\ end but relate 1 to 1 with the source code blocks.

\ For more info see Mr. La Quey's paper when the FORML
\ proceedings become available.
\    -Scott Squires    GEnie S.W.SQUIRES

\ JForth'ed 22 jan 88 jack woehr
\ jax@well.UUCP well!jax@lll-crg.arpa JAX on GEnie
\ Original file NEURAL-NET.TXT in FIG Software Library
\ was buggy, see {  CM  } and  { CONNECTOR } for comparison.

\ ( Notes on the code                         rel  11/01/87)

\ Because the weights are equal for each input the problem 
\ can be scaled so that no multiplication is required.  Thus 
\ the ON value is given by the max number allowed divided by 
\ I/L.  For purposes of this simulation I have built a byte 
\ oriented simulator with I/L = 8 so that ON = 32 ( 256/8 ).  

\ I suspect that the choice of equal weights does not 
\ involve any true loss of generality.  By connecting an 
\ output to several computing elements we can obtain results 
\ equivalent to the use of weighted inputs.  

\ The connection matrix contains an element# for each input 
\ of each element.  I use byte cells and hence have allowed 
\ for 256 elements.  The connection matrix uses #L I/L * = 
\ 2048 bytes.  

\ ( Notes on the code                         rel  11/01/87)

\ One quickly becomes impressed with how slow the IBM PC is.  
\ This type of simulation is computationally intensive.  
\ There are #L I/L * (here 2048) operations per step, i.e.  
\ essentially one operation per connection, in the inner 
\ loop.  Thus one quickly loses interactivity and those of 
\ us with short spans of attention lose interest.  

\ The Chuck Chip is fast enough to allow useful experiments.  

\ JAXNOTE ( 22 jan 88) so is the 68000 ... JForthified this day!
\ Think that's all I'm going to play with this for this week.

\ MOD: Phil Burk 7/14/89 Added DECIMAL at beginning
\      some more comments, ANEW, .NET.CONTROL added to .NET
\      fixed stack problem in GM.

decimal
INCLUDE? CHOOSE JU:RANDOM

ANEW TASK-J4NEURAL

( Neural Network Data Structure             rel 04/02/87 )

  64		CONSTANT #L          ( # of elements         )
   8		CONSTANT I/L         ( Input/element         )
 256 I/L /	CONSTANT NON         ( Neuron has fired      )
   0		CONSTANT NOFF        ( Neuron has not fired  )
   3		CONSTANT THOLD       ( # inputs reqd to fire )
THOLD NON *	CONSTANT GAMMA

VARIABLE STEP#	\ iteration counter

CREATE >CM   #L  I/L * ALLOT     ( Connect Matrix --- fill with values yerself!)

: CM  ( el# in# -- addr )  \ offset + addr in Connection Matrix array
	SWAP I/L * + >CM + \ last plus was missing in the original!
;


( Neural Network Data Structure            rel 04/02/87 )

VARIABLE >OLD	\ holds addr of current "input" ( not FORTH input) buffer
VARIABLE >NEW	\ holds addr of current "output" buffer ... not my terminology!
CREATE BUF0 #L ALLOT	\ create two buffers which will swap ...
CREATE BUF1 #L ALLOT	\ ... functions every interation

: START  ( -- )  \ init counters & arrays
  STEP# OFF	\ start counter
  BUF0 >OLD !	\ assign input buffer
  BUF1 >NEW !	\ assign output buffer
;

START	\ init this part of system

: NEXT  ( -- )	\ Swap I/O Buffers
	1 STEP# +!	\ increment counter
	>OLD @		\ get addr of previous input buffer
	>NEW @		\ "   "    "  "       output "
	>OLD !		\ and reverse...
	>NEW !		\ ... the buffer assignments
;

: INPUT  ( el# -- addr ) \ add array address to offset
	>OLD @ +
;

: OUTPUT ( el# -- addr ) \ add array address to offset
	>NEW @ +
;

: QUIET ( ---) \ clear the working bytes in both buffers
	8 INPUT  #L 8 - ERASE	\ erase 56 bytes of "input"  buffer
	8 OUTPUT #L 8 - ERASE	\ erase 56 bytes of "output" buffer
;

( Neural Network Simulator                  rel 04/02/87 )

: CONNECT     ( el# el# in# -- )	\ fill in a byte entry in Connect Matrix
	CM C!
;

: DISCONNECT  ( el# in# --     )	\ negate a byte entry in Connect Matrix
	CM DUP
	C@ NEGATE
	SWAP C!
;

: CONNECTION  ( el# in# -- el# )	\ inspect a byte entry in Connect Matrix
	CM C@
;

DEFER DISPLAY   ' NOOP IS DISPLAY

: NEURUN  ( el# -- )   ( Runs a single neuron )
	0 SWAP		\ dummy entry, then bring element number to top
	I/L 0		\ do for input-per-element number of iterations
	DO
		DUP	\ the element number ( --- 0 el# el#)
		I CONNECTION	\ get matrix entry ( --- 0 el# entry)
		INPUT	\ use as index to the current "input" buff ( - 0 el# ad)
		C@	\ fetch value there ( --- 0 el# value)
		ROT + SWAP	\ ( --- value el#)
	LOOP
	SWAP		\ cumulative value to top ( --- el# value)
	GAMMA <		\ within threshold ? 	  ( --- el# t|f)
	IF
		NON
	ELSE
		NOFF
	THEN
	SWAP		\ ( val el# ---)
	OUTPUT C! 	\ put in current output buffer ( ---)
;

: NEURUNS	( Runs the Neural Network )
	#L 8
	DO
		I NEURUN
	LOOP
;

: THINK ( #steps -- )
	QUIET	\ clear inp/output buffers
	0
	DO
		NEURUNS
		DISPLAY
		NEXT 
   LOOP
;


( Neural Network  Network Interface            rel 04/02/87 )

CREATE BIT
   1 ,   2 ,   4 ,   8 ,   16 ,   32 ,   64 ,    128 ,

: IS-NET-INPUT  ( ch -- )
	8 0
	DO
		DUP
		7 I - 2*
		BIT + @
		AND
		IF
			NON
		ELSE
			NOFF
		THEN
		DUP
 		I INPUT C!
		I OUTPUT C!
	LOOP
	DROP
;

: NET-OUTPUT-IS  ( -- ch )
	0
	8 0
	DO
		#L 8 -
		I + OUTPUT C@
		NON =
		IF
			7 I - 2*
			BIT + C@
			+
		THEN
	LOOP
;

( Neural Network  Connection Matrix            rel 04/02/87 )

: GM	( -- , connect neurons for demo. GM for Grey Matter )
\ Connect internal neurons except for output.
	#L 8 - 8
	DO
\ Connect 1/2 of the inputs of each neural element
\ to one of the 8 inputs from the outside world.
		I I/L
		2/ 0
        	DO  ( -- K , i from outer loop )
			DUP
			8 CHOOSE ( -- K K rand )
			SWAP I   ( -- K rand K I )
			CONNECT
		LOOP
\ Connect the other half of the inputs of each neural
\ element to one of the other previous neural elements.
		I/L I/L 2/
        	DO
			DUP
			DUP CHOOSE
			SWAP I
			CONNECT
		LOOP
		DROP
    	LOOP
\ Connect output neurons
    	#L DUP 8 -
    	DO
		I I/L 0
		DO
			DUP
			#L CHOOSE
			SWAP I
			CONNECT
		LOOP
		DROP
	LOOP
;

\ Neural Network  Portable Character Display rel 04/02/87 )

: ?CR ( n -- ) 16 MOD 0= IF CR THEN ;
: .ON ( n -- ) NON = IF ASCII 1 ELSE ASCII 0 THEN EMIT ;

: .STATE ( limit first )
    DO I ?CR I OUTPUT C@ .ON LOOP ;

: .NET-INPUT  CR ." Input: " 8        0 .STATE ;
: .NET-OUTPUT CR ." Output:" cr #L DUP 8 - .STATE ;
: .NETWORK    CR #L       0 .STATE ;

: .CON ( el# -- ) CR DUP . CR I/L 0
     DO DUP  I CM C@ 4 .R  SPACE LOOP DROP ;
: .CONS CR ." Connections " #L 0 DO I .CON LOOP ;

: .NET.CONTROL ( -- , allow interactive control )
    cr cr ." Hit SPACE BAR to continue, '?' for help:" key cr
    dup ascii A >=
    IF $ 20 or  \ convert to lower case
    THEN
    CASE
	ascii q OF ABORT ENDOF
	ascii r OF 256 choose is-net-input ENDOF
	ascii c OF gm ENDOF
	ascii ? OF ." 'q' to quit" cr
		   ." 'r' for new random input" cr
		   ." 'c' to reconnect neurons" cr
		   key drop
                ENDOF
    ENDCASE
    cls \ clear screen
;

: .NET ( -- , display network status as 1s and 0s. )
    .net.control  ( added by PLB )
    CR STEP# @ . CR .NET-INPUT CR .NETWORK CR .NET-OUTPUT
;

: DEMO.NEURAL  ( -- , demonstrate neural network )
    cls
    start ( initialize neural system )
    ' .net is display
    gm
    256 choose dup ." Input is (hex) " .hex cr
    is-net-input
    100 think
;


\ -------------------------------------------
\ Open a RAW window for single keystroke interaction.
variable NN-WINDOW
variable NN-OLDWINDOW

: NN.OPENWINDOW  ( -- , open a RAW window )
    NN-window @ 0=
    IF  " RAW:0/20/360/170/Neural Net"
        $fopen ?dup
        IF  consoleout @ NN-oldwindow !
            dup NN-window !
            console!
        ELSE ." Warning! - could not open RAW window!" cr
        THEN
    ELSE ." Window already open!" cr
    THEN
;

: NN.CLOSEWINDOW  ( -- , close window if open )
    NN-window @ ?dup
    IF fclose
        NN-oldwindow @ console!
        NN-window off
    THEN
;

: NN.ABORT ( -- )
    nn.closewindow
    ' (abort) is abort
    abort
;

: CLONE.NEURAL  ( -- )
    nn.openwindow
    ' nn.abort is abort
    demo.neural
    nn.closewindow
;
\ --------------------------------------------


\ ( Neural Network Data Structure             rel 04/02/87 )

\  #L is the number of computing elements (neurons) in the 
\  network. Each computing element has I/L inputs/element.  

\  The output of the computing element is scaled so the product 
\  I/L NONN * = max range = 256 here.  

\  THOLD is the threshold number of inputs.  When more than  
\  THOLD inputs are = NONN the computing element fires ...  and 
\  sets its output to = NONN.  

\  The Connection Matrix describes the network.  For each input 
\  of each computing element of the Connection Matrix contains 
\  the number of the computing element connected to that input.  

\ ( Neural Network Data Structure            rel 04/02/87 )

\ The buffers BUF0 and BUF1 contain the current and previous
\ values of the computing element outputs.  At each step in the
\ computation the current output is used as input, the previous
\ output is overwritten, and then the buffers are swapped in
\ preparation for the NEXT step.

\ QUIET puts zeros (NOFF) into the holding registers.  This
\ corresponds to a quiescent network with no initial thoughts.
\ Perhaps this is an example of the ZEN notion of NO MIND.

\ Exercise for the student:

\ What is the sound of one neuron clapping?

( Neural Network Simulator                  rel 04/02/87 )

\ We will need to CONNECT elements to specific inputs.  Also to 
\ DISCONNECT them.  (Note DISCONNECT DISCONNECT does nothing) 
\ Given an element# and an input# CONNECTION returns the 
\ element# that is connected to that specific input.  

\ NEURUN is the basic simulator.  It simply evaluates the total
\ input weight by summing the inputs and compares it with the
\ threshold. If the threshold is exceeded NON is put into the
\ output register else NOFF is placed in the output register.

\ NEURUNS just does it for all of the neurons except for 0 thru 
\ 7 which are reserved to input data from the external world.  

\ THINK begins from a state of NO MIND and computes.

( Neural Network  Network Interface            rel 04/02/87 )

\ IS-NET-INPUT  ( ch -- )
\ maps an 8 bit item on the stack from the world external to the 
\ Neural Network onto the holding registers for the computing 
\ elements numbered from 0 to 7.  Thus as far as the network is 
\ concerned the outside world just looks like the output of 
\ neurons 0 thru 7.  These elements do not change state as the 
\ network computes but simply hold the input character.  

\ NET-OUTPUT-IS  ( -- ch )
\ maps the output of the last 8 computing elements (neurons)
\ onto the stack.

( Neural Network  Connection Matrix            rel 04/02/87 )

\ This screen shows one of an infinity of ways of specifying a
\ Connection Matrix.

\ For this particular choice:

\     a) inputs 0 thru 3 are driven by randomly chosen elements
\        from the outside world.

\     b) inputs 4 thru 7 are driven by randomly chosen elements
\        from the internal world by the network.

\ Each element is connected to the outside world (like a 
\ retina?) and also to the internal world of the network.  

( Neural Network  Display                       rel 04/02/87 )
\ These names provide a simple display system which should be
\ portable to a wide variety of computers.

