\ ( Networks of Neurons Robert E. La Quey FORML 87 ) \ The following is source code presented by Robert La Quey \ at the FORML conference in November, 1987. This was \ released into the public domain so that people could become \ familar with neural nets and how they could be done. \ I've written it up in block sections groupings as it was \ presented in the paper. The shadow blocks are included at the \ end but relate 1 to 1 with the source code blocks. \ For more info see Mr. La Quey's paper when the FORML \ proceedings become available. \ -Scott Squires GEnie S.W.SQUIRES \ JForth'ed 22 jan 88 jack woehr \ jax@well.UUCP well!jax@lll-crg.arpa JAX on GEnie \ Original file NEURAL-NET.TXT in FIG Software Library \ was buggy, see { CM } and { CONNECTOR } for comparison. \ ( Notes on the code rel 11/01/87) \ Because the weights are equal for each input the problem \ can be scaled so that no multiplication is required. Thus \ the ON value is given by the max number allowed divided by \ I/L. For purposes of this simulation I have built a byte \ oriented simulator with I/L = 8 so that ON = 32 ( 256/8 ). \ I suspect that the choice of equal weights does not \ involve any true loss of generality. By connecting an \ output to several computing elements we can obtain results \ equivalent to the use of weighted inputs. \ The connection matrix contains an element# for each input \ of each element. I use byte cells and hence have allowed \ for 256 elements. The connection matrix uses #L I/L * = \ 2048 bytes. \ ( Notes on the code rel 11/01/87) \ One quickly becomes impressed with how slow the IBM PC is. \ This type of simulation is computationally intensive. \ There are #L I/L * (here 2048) operations per step, i.e. \ essentially one operation per connection, in the inner \ loop. Thus one quickly loses interactivity and those of \ us with short spans of attention lose interest. \ The Chuck Chip is fast enough to allow useful experiments. \ JAXNOTE ( 22 jan 88) so is the 68000 ... JForthified this day! \ Think that's all I'm going to play with this for this week. \ MOD: Phil Burk 7/14/89 Added DECIMAL at beginning \ some more comments, ANEW, .NET.CONTROL added to .NET \ fixed stack problem in GM. decimal INCLUDE? CHOOSE JU:RANDOM ANEW TASK-J4NEURAL ( Neural Network Data Structure rel 04/02/87 ) 64 CONSTANT #L ( # of elements ) 8 CONSTANT I/L ( Input/element ) 256 I/L / CONSTANT NON ( Neuron has fired ) 0 CONSTANT NOFF ( Neuron has not fired ) 3 CONSTANT THOLD ( # inputs reqd to fire ) THOLD NON * CONSTANT GAMMA VARIABLE STEP# \ iteration counter CREATE >CM #L I/L * ALLOT ( Connect Matrix --- fill with values yerself!) : CM ( el# in# -- addr ) \ offset + addr in Connection Matrix array SWAP I/L * + >CM + \ last plus was missing in the original! ; ( Neural Network Data Structure rel 04/02/87 ) VARIABLE >OLD \ holds addr of current "input" ( not FORTH input) buffer VARIABLE >NEW \ holds addr of current "output" buffer ... not my terminology! CREATE BUF0 #L ALLOT \ create two buffers which will swap ... CREATE BUF1 #L ALLOT \ ... functions every interation : START ( -- ) \ init counters & arrays STEP# OFF \ start counter BUF0 >OLD ! \ assign input buffer BUF1 >NEW ! \ assign output buffer ; START \ init this part of system : NEXT ( -- ) \ Swap I/O Buffers 1 STEP# +! \ increment counter >OLD @ \ get addr of previous input buffer >NEW @ \ " " " " output " >OLD ! \ and reverse... >NEW ! \ ... the buffer assignments ; : INPUT ( el# -- addr ) \ add array address to offset >OLD @ + ; : OUTPUT ( el# -- addr ) \ add array address to offset >NEW @ + ; : QUIET ( ---) \ clear the working bytes in both buffers 8 INPUT #L 8 - ERASE \ erase 56 bytes of "input" buffer 8 OUTPUT #L 8 - ERASE \ erase 56 bytes of "output" buffer ; ( Neural Network Simulator rel 04/02/87 ) : CONNECT ( el# el# in# -- ) \ fill in a byte entry in Connect Matrix CM C! ; : DISCONNECT ( el# in# -- ) \ negate a byte entry in Connect Matrix CM DUP C@ NEGATE SWAP C! ; : CONNECTION ( el# in# -- el# ) \ inspect a byte entry in Connect Matrix CM C@ ; DEFER DISPLAY ' NOOP IS DISPLAY : NEURUN ( el# -- ) ( Runs a single neuron ) 0 SWAP \ dummy entry, then bring element number to top I/L 0 \ do for input-per-element number of iterations DO DUP \ the element number ( --- 0 el# el#) I CONNECTION \ get matrix entry ( --- 0 el# entry) INPUT \ use as index to the current "input" buff ( - 0 el# ad) C@ \ fetch value there ( --- 0 el# value) ROT + SWAP \ ( --- value el#) LOOP SWAP \ cumulative value to top ( --- el# value) GAMMA < \ within threshold ? ( --- el# t|f) IF NON ELSE NOFF THEN SWAP \ ( val el# ---) OUTPUT C! \ put in current output buffer ( ---) ; : NEURUNS ( Runs the Neural Network ) #L 8 DO I NEURUN LOOP ; : THINK ( #steps -- ) QUIET \ clear inp/output buffers 0 DO NEURUNS DISPLAY NEXT LOOP ; ( Neural Network Network Interface rel 04/02/87 ) CREATE BIT 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , : IS-NET-INPUT ( ch -- ) 8 0 DO DUP 7 I - 2* BIT + @ AND IF NON ELSE NOFF THEN DUP I INPUT C! I OUTPUT C! LOOP DROP ; : NET-OUTPUT-IS ( -- ch ) 0 8 0 DO #L 8 - I + OUTPUT C@ NON = IF 7 I - 2* BIT + C@ + THEN LOOP ; ( Neural Network Connection Matrix rel 04/02/87 ) : GM ( -- , connect neurons for demo. GM for Grey Matter ) \ Connect internal neurons except for output. #L 8 - 8 DO \ Connect 1/2 of the inputs of each neural element \ to one of the 8 inputs from the outside world. I I/L 2/ 0 DO ( -- K , i from outer loop ) DUP 8 CHOOSE ( -- K K rand ) SWAP I ( -- K rand K I ) CONNECT LOOP \ Connect the other half of the inputs of each neural \ element to one of the other previous neural elements. I/L I/L 2/ DO DUP DUP CHOOSE SWAP I CONNECT LOOP DROP LOOP \ Connect output neurons #L DUP 8 - DO I I/L 0 DO DUP #L CHOOSE SWAP I CONNECT LOOP DROP LOOP ; \ Neural Network Portable Character Display rel 04/02/87 ) : ?CR ( n -- ) 16 MOD 0= IF CR THEN ; : .ON ( n -- ) NON = IF ASCII 1 ELSE ASCII 0 THEN EMIT ; : .STATE ( limit first ) DO I ?CR I OUTPUT C@ .ON LOOP ; : .NET-INPUT CR ." Input: " 8 0 .STATE ; : .NET-OUTPUT CR ." Output:" cr #L DUP 8 - .STATE ; : .NETWORK CR #L 0 .STATE ; : .CON ( el# -- ) CR DUP . CR I/L 0 DO DUP I CM C@ 4 .R SPACE LOOP DROP ; : .CONS CR ." Connections " #L 0 DO I .CON LOOP ; : .NET.CONTROL ( -- , allow interactive control ) cr cr ." Hit SPACE BAR to continue, '?' for help:" key cr dup ascii A >= IF $ 20 or \ convert to lower case THEN CASE ascii q OF ABORT ENDOF ascii r OF 256 choose is-net-input ENDOF ascii c OF gm ENDOF ascii ? OF ." 'q' to quit" cr ." 'r' for new random input" cr ." 'c' to reconnect neurons" cr key drop ENDOF ENDCASE cls \ clear screen ; : .NET ( -- , display network status as 1s and 0s. ) .net.control ( added by PLB ) CR STEP# @ . CR .NET-INPUT CR .NETWORK CR .NET-OUTPUT ; : DEMO.NEURAL ( -- , demonstrate neural network ) cls start ( initialize neural system ) ' .net is display gm 256 choose dup ." Input is (hex) " .hex cr is-net-input 100 think ; \ ------------------------------------------- \ Open a RAW window for single keystroke interaction. variable NN-WINDOW variable NN-OLDWINDOW : NN.OPENWINDOW ( -- , open a RAW window ) NN-window @ 0= IF " RAW:0/20/360/170/Neural Net" $fopen ?dup IF consoleout @ NN-oldwindow ! dup NN-window ! console! ELSE ." Warning! - could not open RAW window!" cr THEN ELSE ." Window already open!" cr THEN ; : NN.CLOSEWINDOW ( -- , close window if open ) NN-window @ ?dup IF fclose NN-oldwindow @ console! NN-window off THEN ; : NN.ABORT ( -- ) nn.closewindow ' (abort) is abort abort ; : CLONE.NEURAL ( -- ) nn.openwindow ' nn.abort is abort demo.neural nn.closewindow ; \ -------------------------------------------- \ ( Neural Network Data Structure rel 04/02/87 ) \ #L is the number of computing elements (neurons) in the \ network. Each computing element has I/L inputs/element. \ The output of the computing element is scaled so the product \ I/L NONN * = max range = 256 here. \ THOLD is the threshold number of inputs. When more than \ THOLD inputs are = NONN the computing element fires ... and \ sets its output to = NONN. \ The Connection Matrix describes the network. For each input \ of each computing element of the Connection Matrix contains \ the number of the computing element connected to that input. \ ( Neural Network Data Structure rel 04/02/87 ) \ The buffers BUF0 and BUF1 contain the current and previous \ values of the computing element outputs. At each step in the \ computation the current output is used as input, the previous \ output is overwritten, and then the buffers are swapped in \ preparation for the NEXT step. \ QUIET puts zeros (NOFF) into the holding registers. This \ corresponds to a quiescent network with no initial thoughts. \ Perhaps this is an example of the ZEN notion of NO MIND. \ Exercise for the student: \ What is the sound of one neuron clapping? ( Neural Network Simulator rel 04/02/87 ) \ We will need to CONNECT elements to specific inputs. Also to \ DISCONNECT them. (Note DISCONNECT DISCONNECT does nothing) \ Given an element# and an input# CONNECTION returns the \ element# that is connected to that specific input. \ NEURUN is the basic simulator. It simply evaluates the total \ input weight by summing the inputs and compares it with the \ threshold. If the threshold is exceeded NON is put into the \ output register else NOFF is placed in the output register. \ NEURUNS just does it for all of the neurons except for 0 thru \ 7 which are reserved to input data from the external world. \ THINK begins from a state of NO MIND and computes. ( Neural Network Network Interface rel 04/02/87 ) \ IS-NET-INPUT ( ch -- ) \ maps an 8 bit item on the stack from the world external to the \ Neural Network onto the holding registers for the computing \ elements numbered from 0 to 7. Thus as far as the network is \ concerned the outside world just looks like the output of \ neurons 0 thru 7. These elements do not change state as the \ network computes but simply hold the input character. \ NET-OUTPUT-IS ( -- ch ) \ maps the output of the last 8 computing elements (neurons) \ onto the stack. ( Neural Network Connection Matrix rel 04/02/87 ) \ This screen shows one of an infinity of ways of specifying a \ Connection Matrix. \ For this particular choice: \ a) inputs 0 thru 3 are driven by randomly chosen elements \ from the outside world. \ b) inputs 4 thru 7 are driven by randomly chosen elements \ from the internal world by the network. \ Each element is connected to the outside world (like a \ retina?) and also to the internal world of the network. ( Neural Network Display rel 04/02/87 ) \ These names provide a simple display system which should be \ portable to a wide variety of computers.