Subject: changes to "comp.ai.neural-nets FAQ" -- monthly posting
Supersedes: <nn.changes.posting_825566444@hotellng.unx.sas.com>
Date: Fri, 29 Mar 1996 04:00:40 GMT

==> nn1.changes.body <==
*** nn1.oldbody.Wed Feb 28 23:00:21 1996
--- nn1.body.Thu Mar 28 23:00:07 1996
***************
*** 1,8 ****
  Archive-name: ai-faq/neural-nets/part1
! Last-modified: 1996-02-16
  URL: ftp://ftp.sas.com/pub/neural/FAQ.html
  Maintainer: saswss@unx.sas.com (Warren S. Sarle)
  
- 
    ---------------------------------------------------------------
      Additions, corrections, or improvements are always welcome.
--- 1,7 ----
  Archive-name: ai-faq/neural-nets/part1
! Last-modified: 1996-03-27
  URL: ftp://ftp.sas.com/pub/neural/FAQ.html
  Maintainer: saswss@unx.sas.com (Warren S. Sarle)
  
    ---------------------------------------------------------------
      Additions, corrections, or improvements are always welcome.
***************
*** 13,17 ****
    ---------------------------------------------------------------
  
- 
  This is the first of seven parts of a monthly posting to the Usenet
  newsgroup comp.ai.neural-nets (as well as comp.answers and news.answers,
--- 12,15 ----
***************
*** 61,80 ****
     What is this newsgroup for? How shall it be used?
     What is a neural network (NN)?
!    What can you do with a Neural Network and what not?
!    Who is concerned with Neural Networks?
  
  Part 2: Learning
  
-    What does 'backprop' mean? What is 'overfitting'?
-    Why use a bias input? Why activation functions?
-    How many hidden units should I use?
     How many learning methods for NNs exist? Which?
     What about Genetic Algorithms and Evolutionary Computation?
     What about Fuzzy Logic?
-    How are NNs related to statistical methods?
  
  Part 3: Information resources
  
!    Good introductory literature about Neural Networks?
     Any journals and magazines about Neural Networks?
     The most important conferences concerned with Neural Networks?
--- 59,94 ----
     What is this newsgroup for? How shall it be used?
     What is a neural network (NN)?
!    What can you do with an NN and what not?
!    Who is concerned with NNs?
!    How are layers counted?
!    How are NNs related to statistical methods?
  
  Part 2: Learning
  
     How many learning methods for NNs exist? Which?
+    What is backprop?
+    What are conjugate gradients, Levenberg-Marquardt, etc.?
+    How should categories be coded?
+    Why use a bias input?
+    Why use activation functions?
+    What is a softmax activation function?
+    What is overfitting and how can I avoid it?
+    What is jitter? (Training with noise)
+    What is early stopping?
+    What is weight decay?
+    What is Bayesian estimation?
+    How many hidden units should I use?
+    How can generalization error be estimated?
+    What are cross-validation and bootstrapping?
+    Should I normalize/standardize/rescale the data?
+    What is ART?
+    What is PNN?
+    What is GRNN?
     What about Genetic Algorithms and Evolutionary Computation?
     What about Fuzzy Logic?
  
  Part 3: Information resources
  
!    Good literature about Neural Networks?
     Any journals and magazines about Neural Networks?
     The most important conferences concerned with Neural Networks?
***************
*** 94,100 ****
     Commercial software packages for NN simulation?
  
! Part 7: Hardware
  
     Neural Network hardware?
  
  ------------------------------------------------------------------------
--- 108,115 ----
     Commercial software packages for NN simulation?
  
! Part 7: Hardware, etc.
  
     Neural Network hardware?
+    Unanswered FAQs
  
  ------------------------------------------------------------------------
***************
*** 237,295 ****
  First of all, when we are talking about a neural network, we should more
  properly say "artificial neural network" (ANN), because that is what we mean
! most of the time. Biological neural networks are much more complicated in
! their elementary structures than the mathematical models we use for ANNs.
  
- A vague description is as follows:
- 
- An ANN is a network of many simple processors ("units"), each possibly
- having a small amount of local memory. The units are connected by
- unidirectional communication channels ("connections"), which usually carry
- numeric (as opposed to symbolic) data. The units operate only on their local
- data and on the inputs they receive via the connections.
- 
- The design motivation is what distinguishes neural networks from other
- mathematical techniques:
- 
- A neural network is a processing device, either an algorithm, or actual
- hardware, whose design was motivated by the design and functioning of human
- brains and components thereof.
- 
- Most neural networks have some sort of "training" rule whereby the weights
- of connections are adjusted on the basis of data. In other words, neural
- networks "learn" from examples, just like children learn to recognize dogs
- from examples of dogs, and exhibit some structural capability for
- generalization.
- 
- Neural networks normally have great potential for parallelism, since the
- computations of the components are independent of each other. Some people
- regard massive parallelism and high connectivity to be defining
- characteristics of neural networks, but such requirements rule out various
- simple models, such as simple linear regression, which are usefully regarded
- as special cases of neural networks. 
- 
  ------------------------------------------------------------------------
  
! Subject: What can you do with a Neural Network and
! ==================================================
! what not?
! =========
  
  In principle, NNs can compute any computable function, i.e. they can do
! everything a normal digital computer can do. Especially anything that can be
! represented as a mapping between vector spaces can be approximated to
! arbitrary precision by feedforward NNs (which is the most often used type).
! 
! In practice, NNs are especially useful for function approximation/mapping
! problems which are tolerant of some errors, have lots of example data
! available, but to which hard and fast rules cannot easily be applied. NNs
! are, at least today, difficult to apply successfully to problems that
! concern manipulation of symbols and memory. 
  
  ------------------------------------------------------------------------
  
! Subject: Who is concerned with Neural Networks?
! ===============================================
  
! Neural Networks are interesting for quite a lot of very dissimilar people: 
  
   o Computer scientists want to find out about the properties of non-symbolic
--- 252,374 ----
  First of all, when we are talking about a neural network, we should more
  properly say "artificial neural network" (ANN), because that is what we mean
! most of the time in comp.ai.neural-nets. Biological neural networks are much
! more complicated than the mathematical models we use for ANNs. But it is
! customary to be lazy and drop the "A" or the "artificial". 
! 
! There is no universally accepted definition of an NN. But perhaps most
! people in the field would agree that an NN is a network of many simple
! processors ("units"), each possibly having a small amount of local memory.
! The units are connected by communication channels ("connections") which
! usually carry numeric (as opposed to symbolic) data, encoded by any of
! various means. The units operate only on their local data and on the inputs
! they receive via the connections. The restriction to local operations is
! often relaxed during training. 
! 
! Some NNs are models of biological neural networks and some are not, but
! historically, much of the inspiration for the field of NNs came from the
! desire to produce artificial systems capable of sophisticated, perhaps
! "intelligent", computations similar to those that the human brain routinely
! performs, and thereby possibly to enhance our understanding of the human
! brain. 
! 
! Most NNs have some sort of "training" rule whereby the weights of
! connections are adjusted on the basis of data. In other words, NNs "learn"
! from examples (as children learn to recognize dogs from examples of dogs)
! and exhibit some capability for generalization beyond the training data. 
! 
! NNs normally have great potential for parallelism, since the computations of
! the components are largely independent of each other. Some people regard
! massive parallelism and high connectivity to be defining characteristics of
! NNs, but such requirements rule out various simple models, such as simple
! linear regression (a minimal feedforward net with only two units plus bias),
! which are usefully regarded as special cases of NNs. 
! 
! Here is a sampling of definitions from the books on the FAQ maintainer's
! shelf. None will please everyone. Perhaps for that reason many NN textbooks
! do not explicitly define neural networks. 
! 
! According to the DARPA Neural Network Study (1988, AFCEA International
! Press, p. 60): 
! 
!    ... a neural network is a system composed of many simple processing
!    elements operating in parallel whose function is determined by
!    network structure, connection strengths, and the processing performed
!    at computing elements or nodes. 
! 
! According to Haykin, S. (1994), Neural Networks: A Comprehensive
! Foundation, NY: Macmillan, p. 2: 
! 
!    A neural network is a massively parallel distributed processor that
!    has a natural propensity for storing experiential knowledge and
!    making it available for use. It resembles the brain in two respects: 
! 
!    1. Knowledge is acquired by the network through a learning process. 
!    2. Interneuron connection strengths known as synaptic weights are
!       used to store the knowledge. 
! 
! According to Nigrin, A. (1993), Neural Networks for Pattern Recognition,
! Cambridge, MA: The MIT Press, p. 11: 
! 
!    A neural network is a circuit composed of a very large number of
!    simple processing elements that are neurally based. Each element
!    operates only on local information. Furthermore each element operates
!    asynchronously; thus there is no overall system clock. 
! 
! According to Zurada, J.M. (1992), Introduction To Artificial Neural Systems,
! Boston: PWS Publishing Company, p. xv: 
! 
!    Artificial neural systems, or neural networks, are physical cellular
!    systems which can acquire, store, and utilize experiential knowledge.
! 
! For more information on "What is a neural network?", with examples and
! diagrams, see Leslie S. Smith's on-line introduction at: 
! http://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html. 
  
  ------------------------------------------------------------------------
  
! Subject: What can you do with an NN and what not?
! =================================================
  
  In principle, NNs can compute any computable function, i.e. they can do
! everything a normal digital computer can do. 
! 
! In practice, NNs are especially useful for classification and function
! approximation/mapping problems which are tolerant of some imprecision, which
! have lots of training data available, but to which hard and fast rules (such
! as those that might be used in an expert system) cannot easily be applied.
! Almost any mapping between vector spaces can be approximated to arbitrary
! precision by feedforward NNs (which are the type most often used in
! practical applications) if you have enough data and enough computing
! resources. 
! 
! NNs are, at least today, difficult to apply successfully to problems that
! concern manipulation of symbols and memory. And there are no methods for
! training NNs that can magically create information that is not contained in
! the training data. 
! 
! For examples of NN applications, see: 
! 
!  o The Pacific Northwest National Laboratory web page at 
!    http://www.emsl.pnl.gov:2080/docs/cie/neural/neural.research.html. 
!  o The Stimulation Initiative for European Neural Applications web page at 
!    http://www.mbfys.kun.nl/snn/siena/cases/ 
!  o The DTI NeuroComputing Web's Applications Portfolio at 
!    http://www.globalweb.co.uk/nctt/portfolio/. 
!  o The Applications Corner, provided by NeuroDimension, Inc., at 
!    http://www.nd.com/appcornr/purpose.htm. 
!  o Chen, C.H., ed. (1996) Fuzzy Logic and Neural Network Handbook, NY:
!    McGraw-Hill, ISBN 0-07-011189-8. 
!  o Trippi, R.R. & Turban, E. (1993), Neural Networks in Finance and
!    Investing, Chicago: Probus, ISBN 1-55738-452-5. 
!  o The series Advances in Neural Information Processing Systems containing
!    proceedings of the conference of the same name, published yearly by
!    Morgan Kauffman starting in 1989. 
  
  ------------------------------------------------------------------------
  
! Subject: Who is concerned with NNs?
! ===================================
  
! Neural Networks are interesting for quite a lot of very different people: 
  
   o Computer scientists want to find out about the properties of non-symbolic
***************
*** 309,312 ****
--- 388,552 ----
   o Philosophers and some other people may also be interested in Neural
     Networks for various reasons. 
+ 
+ For world-wide lists of groups doing research on NNs, see the Foundation for
+ Neural Networks's (SNN) page at 
+ http://www.mbfys.kun.nl/snn/pointers/groups.html and see Neural Networks
+ Research on the IEEE Neural Network Council's homepage 
+ http://www.ieee.org/nnc. 
+ 
+ ------------------------------------------------------------------------
+ 
+ Subject: How are layers counted? 
+ =================================
+ 
+ This is a matter of considerable dispute. 
+ 
+  o Some people count layers of units. But of these people, some count the
+    input layer and some don't. 
+ 
+  o Some people count layers of weights. But I have no idea how they count
+    skip-layer connections. 
+ 
+ To avoid ambiguity, you should speak of a 2-hidden-layer network, not a
+ 4-layer network (as some would call it) or 3-layer network (as others would
+ call it). And if the connections follow any pattern other than fully
+ connecting each layer to the next and to no others, you should carefully
+ specify the connections. 
+ 
+ ------------------------------------------------------------------------
+ 
+ Subject: How are NNs related to statistical methods? 
+ =====================================================
+ 
+ There is considerable overlap between the fields of neural networks and
+ statistics. Statistics is concerned with data analysis. In neural network
+ terminology, statistical inference means learning to generalize from noisy
+ data. Some neural networks are not concerned with data analysis (e.g., those
+ intended to model biological systems) and therefore have little to do with
+ statistics. Some neural networks do not learn (e.g., Hopfield nets) and
+ therefore have little to do with statistics. Some neural networks can learn
+ successfully only from noise-free data (e.g., ART or the perceptron rule)
+ and therefore would not be considered statistical methods. But most neural
+ networks that can learn to generalize effectively from noisy data are
+ similar or identical to statistical methods. For example: 
+ 
+  o Feedforward nets with no hidden layer (including functional-link neural
+    nets and higher-order neural nets) are basically generalized linear
+    models. 
+  o Feedforward nets with one hidden layer are closely related to projection
+    pursuit regression. 
+  o Probabilistic neural nets are identical to kernel discriminant analysis. 
+  o Kohonen nets for adaptive vector quantization are very similar to k-means
+    cluster analysis. 
+  o Hebbian learning is closely related to principal component analysis. 
+ 
+ Some neural network areas that appear to have no close relatives in the
+ existing statistical literature are: 
+ 
+  o Kohonen's self-organizing maps. 
+  o Reinforcement learning ((although this is treated in the operations
+    research literature as Markov decision processes). 
+  o Stopped training (the purpose and effect of stopped training are similar
+    to shrinkage estimation, but the method is quite different). 
+ 
+ Feedforward nets are a subset of the class of nonlinear regression and
+ discrimination models. Statisticians have studied the properties of this
+ general class but had not considered the specific case of feedforward neural
+ nets before such networks were popularized in the neural network field.
+ Still, many results from the statistical theory of nonlinear models apply
+ directly to feedforward nets, and the methods that are commonly used for
+ fitting nonlinear models, such as various Levenberg-Marquardt and conjugate
+ gradient algorithms, can be used to train feedforward nets. 
+ 
+ While neural nets are often defined in terms of their algorithms or
+ implementations, statistical methods are usually defined in terms of their
+ results. The arithmetic mean, for example, can be computed by a (very
+ simple) backprop net, by applying the usual formula SUM(x_i)/n, or by
+ various other methods. What you get is still an arithmetic mean regardless
+ of how you compute it. So a statistician would consider standard backprop,
+ Quickprop, and Levenberg-Marquardt as different algorithms for implementing
+ the same statistical model such as a feedforward net. On the other hand,
+ different training criteria, such as least squares and cross entropy, are
+ viewed by statisticians as fundamentally different estimation methods with
+ different statistical properties. 
+ 
+ It is sometimes claimed that neural networks, unlike statistical models,
+ require no distributional assumptions. In fact, neural networks involve
+ exactly the same sort of distributional assumptions as statistical models,
+ but statisticians study the consequences and importance of these assumptions
+ while most neural networkers ignore them. For example, least-squares
+ training methods are widely used by statisticians and neural networkers.
+ Statisticians realize that least-squares training involves implicit
+ distributional assumptions in that least-squares estimates have certain
+ optimality properties for noise that is normally distributed with equal
+ variance for all training cases and that is independent between different
+ cases. These optimality properties are consequences of the fact that
+ least-squares estimation is maximum likelihood under those conditions.
+ Similarly, cross-entropy is maximum likelihood for noise with a Bernoulli
+ distribution. If you study the distributional assumptions, then you can
+ recognize and deal with violations of the assumptions. For example, if you
+ have normally distributed noise but some training cases have greater noise
+ variance than others, then you may be able to use weighted least squares
+ instead of ordinary least squares to obtain more efficient estimates. 
+ 
+ Communication between statisticians and neural net researchers is often
+ hindered by the different terminology used in the two fields. There is a
+ comparison of neural net and statistical jargon in 
+ ftp://ftp.sas.com/pub/neural/jargon 
+ 
+ References: 
+ 
+    Balakrishnan, P.V., Cooper, M.C., Jacob, V.S., and Lewis, P.A. (1994) "A
+    study of the classification capabilities of neural networks using
+    unsupervised learning: A comparison with k-means clustering",
+    Psychometrika, 59, 509-525. 
+ 
+    Bishop, C.M. (1995), Neural Networks for Pattern Recognition, Oxford:
+    Oxford University Press. 
+ 
+    Chatfield, C. (1993), "Neural networks: Forecasting breakthrough or
+    passing fad", International Journal of Forecasting, 9, 1-3. 
+ 
+    Cheng, B. and Titterington, D.M. (1994), "Neural Networks: A Review from
+    a Statistical Perspective", Statistical Science, 9, 2-54. 
+ 
+    Geman, S., Bienenstock, E. and Doursat, R. (1992), "Neural Networks and
+    the Bias/Variance Dilemma", Neural Computation, 4, 1-58. 
+ 
+    Kuan, C.-M. and White, H. (1994), "Artificial Neural Networks: An
+    Econometric Perspective", Econometric Reviews, 13, 1-91. 
+ 
+    Kushner, H. & Clark, D. (1978), Stochastic Approximation Methods for
+    Constrained and Unconstrained Systems, Springer-Verlag. 
+ 
+    Michie, D., Spiegelhalter, D.J. and Taylor, C.C. (1994), Machine
+    Learning, Neural and Statistical Classification, Ellis Horwood. 
+ 
+    Ripley, B.D. (1993), "Statistical Aspects of Neural Networks", in O.E.
+    Barndorff-Nielsen, J.L. Jensen and W.S. Kendall, eds., Networks and
+    Chaos: Statistical and Probabilistic Aspects, Chapman & Hall. ISBN 0 412
+    46530 2. 
+ 
+    Ripley, B.D. (1994), "Neural Networks and Related Methods for
+    Classification," Journal of the Royal Statistical Society, Series B, 56,
+    409-456. 
+ 
+    Ripley, B.D. (1996) Pattern Recognition and Neural Networks, Cambridge:
+    Cambridge University Press. 
+ 
+    Sarle, W.S. (1994), "Neural Networks and Statistical Models," 
+    Proceedings of the Nineteenth Annual SAS Users Group International
+    Conference, Cary, NC: SAS Institute, pp 1538-1550. (
+    ftp://ftp.sas.com/pub/neural/neural1.ps) 
+ 
+    White, H. (1989), "Learning in Artificial Neural Networks: A Statistical
+    Perspective," Neural Computation, 1, 425-464. 
+ 
+    White, H. (1989), "Some Asymptotic Results for Learning in Single Hidden
+    Layer Feedforward Network Models", J. of the American Statistical Assoc.,
+    84, 1008-1013. 
+ 
+    White, H. (1992), Artificial Neural Networks: Approximation and Learning
+    Theory, Blackwell. 
  
  ------------------------------------------------------------------------

==> nn2.changes.body <==
*** nn2.oldbody.Wed Feb 28 23:00:24 1996
--- nn2.body.Thu Mar 28 23:00:12 1996
***************
*** 1,4 ****
  Archive-name: ai-faq/neural-nets/part2
! Last-modified: 1996-02-22
  URL: ftp://ftp.sas.com/pub/neural/FAQ2.html
  Maintainer: saswss@unx.sas.com (Warren S. Sarle)
--- 1,4 ----
  Archive-name: ai-faq/neural-nets/part2
! Last-modified: 1996-03-28
  URL: ftp://ftp.sas.com/pub/neural/FAQ2.html
  Maintainer: saswss@unx.sas.com (Warren S. Sarle)
***************
*** 12,61 ****
  
  Part 1: Introduction
- 
-    What is this newsgroup for? How shall it be used?
-    What is a neural network (NN)?
-    What can you do with a Neural Network and what not?
-    Who is concerned with Neural Networks?
- 
  Part 2: Learning
  
-    What does 'backprop' mean? What is 'overfitting'?
-    Why use a bias input? Why activation functions?
-    How many hidden units should I use?
     How many learning methods for NNs exist? Which?
     What about Genetic Algorithms and Evolutionary Computation?
     What about Fuzzy Logic?
-    How are NNs related to statistical methods?
  
  Part 3: Information resources
- 
-    Good introductory literature about Neural Networks?
-    Any journals and magazines about Neural Networks?
-    The most important conferences concerned with Neural Networks?
-    Neural Network Associations?
-    Other sources of information about NNs?
- 
  Part 4: Datasets
  
!    Databases for experimentation with NNs?
  
! Part 5: Free software
  
!    Freely available software packages for NN simulation?
  
! Part 6: Commercial software
  
!    Commercial software packages for NN simulation?
  
! Part 7: Hardware
  
!    Neural Network hardware?
  
  ------------------------------------------------------------------------
  
! Subject: What does 'backprop' mean? What is
! ===========================================
! 'overfitting'? 
! ===============
  
  Backprop is short for backpropagation of error. The term backpropagation
--- 12,129 ----
  
  Part 1: Introduction
  Part 2: Learning
  
     How many learning methods for NNs exist? Which?
+    What is backprop?
+    What are conjugate gradients, Levenberg-Marquardt, etc.?
+    How should categories be coded?
+    Why use a bias input?
+    Why use activation functions?
+    What is a softmax activation function?
+    What is overfitting and how can I avoid it?
+    What is jitter? (Training with noise)
+    What is early stopping?
+    What is weight decay?
+    What is Bayesian estimation?
+    How many hidden units should I use?
+    How can generalization error be estimated?
+    What are cross-validation and bootstrapping?
+    Should I normalize/standardize/rescale the data?
+    What is ART?
+    What is PNN?
+    What is GRNN?
     What about Genetic Algorithms and Evolutionary Computation?
     What about Fuzzy Logic?
  
  Part 3: Information resources
  Part 4: Datasets
+ Part 5: Free software
+ Part 6: Commercial software
+ Part 7: Hardware
  
! ------------------------------------------------------------------------
  
! Subject: How many learning methods for NNs exist?
! =================================================
! Which?
! ======
  
! There are many many learning methods for NNs by now. Nobody knows exactly
! how many. New ones (or at least variations of existing ones) are invented
! every week. Below is a collection of some of the most well known methods,
! not claiming to be complete.
! 
! The main categorization of these methods is the distinction between
! supervised and unsupervised learning: 
! 
!  o In supervised learning, there is a "teacher" who in the learning phase
!    "tells" the net how well it performs ("reinforcement learning") or what
!    the correct behavior would have been ("fully supervised learning"). 
!  o In unsupervised learning the net is autonomous: it just looks at the data
!    it is presented with, finds out about some of the properties of the data
!    set and learns to reflect these properties in its output. What exactly
!    these properties are, that the network can learn to recognise, depends on
!    the particular network model and learning method. Usually, the net learns
!    some compressed representation of the data. 
  
! Many of these learning methods are closely connected with a certain (class
! of) network topology.
  
! Now here is the list, just giving some names:
  
! 1. UNSUPERVISED LEARNING (i.e. without a "teacher"):
!      1). Feedback Nets:
!         a). Additive Grossberg (AG)
!         b). Shunting Grossberg (SG)
!         c). Binary Adaptive Resonance Theory (ART1)
!         d). Analog Adaptive Resonance Theory (ART2, ART2a)
!         e). Discrete Hopfield (DH)
!         f). Continuous Hopfield (CH)
!         g). Discrete Bidirectional Associative Memory (BAM)
!         h). Temporal Associative Memory (TAM)
!         i). Adaptive Bidirectional Associative Memory (ABAM)
!         j). Kohonen Self-organizing Map/Topology-preserving map (SOM/TPM)
!         k). Competitive learning
!      2). Feedforward-only Nets:
!         a). Learning Matrix (LM)
!         b). Driver-Reinforcement Learning (DR)
!         c). Linear Associative Memory (LAM)
!         d). Optimal Linear Associative Memory (OLAM)
!         e). Sparse Distributed Associative Memory (SDM)
!         f). Fuzzy Associative Memory (FAM)
!         g). Counterprogation (CPN)
  
! 2. SUPERVISED LEARNING (i.e. with a "teacher"):
!      1). Feedback Nets:
!         a). Brain-State-in-a-Box (BSB)
!         b). Fuzzy Congitive Map (FCM)
!         c). Boltzmann Machine (BM)
!         d). Mean Field Annealing (MFT)
!         e). Recurrent Cascade Correlation (RCC)
!         f). Backpropagation through time (BPTT)
!         g). Real-time recurrent learning (RTRL)
!         h). Recurrent Extended Kalman Filter (EKF)
!      2). Feedforward-only Nets:
!         a). Perceptron
!         b). Adaline, Madaline
!         c). Backpropagation (BP)
!         d). Cauchy Machine (CM)
!         e). Adaptive Heuristic Critic (AHC)
!         f). Time Delay Neural Network (TDNN)
!         g). Associative Reward Penalty (ARP)
!         h). Avalanche Matched Filter (AMF)
!         i). Backpercolation (Perc)
!         j). Artmap
!         k). Adaptive Logic Network (ALN)
!         l). Cascade Correlation (CasCor)
!         m). Extended Kalman Filter(EKF)
!         n). Learning Vector Quantization (LVQ)
!         o). Probabilistic Neural Network (PNN)
!         p). General Regression Neural Network (GRNN) 
  
  ------------------------------------------------------------------------
  
! Subject: What is backprop? 
! ===========================
  
  Backprop is short for backpropagation of error. The term backpropagation
***************
*** 63,106 ****
  method for computing the error gradient for a feedforward network, a
  straightforward but elegant application of the chain rule of elementary
! calculus. By extension, backpropagation or backprop refers to a training
! method that uses backpropagation to compute the gradient. By further
! extension, a backprop network is a feedforward network trained by
! backpropagation. Standard backprop is a euphemism for the generalized delta
! rule, the training algorithm that was popularized by Rumelhart, Hinton, and
! Williams in chapter 8 of Rumelhart and McClelland (1986) and that remains
! the most widely used supervised training method for neural nets. 
! 
! Literature:
!    Rumelhart, D. E. and McClelland, J. L. (1986): Parallel Distributed
!    Processing: Explorations in the Microstructure of Cognition (volume 1, pp
!    318-362). The MIT Press. 
! 
! (this is the classic one) or one of the dozens of other books or articles on
! backpropagation (see also question 'literature').
! 
! 'Overfitting' (often also called 'overtraining' or 'overlearning') is the
! phenomenon that in most cases a network gets worse instead of better after a
! certain point during training when it is trained to as low errors as
! possible. This is because such long training may make the network 'memorize'
! the training patterns, including all of their peculiarities. However, one is
! usually interested in the generalization of the network, i.e., the error it
! exhibits on cases NOT seen during training. Learning the peculiarities of
! the training set makes the generalization worse. The network should only
! learn the general structure of the training cases. 
! 
! There are various methods to fight overfitting. The two most important
! classes of such methods are regularization methods (such as weight decay)
! and early stopping. Regularization methods try to limit the complexity of
! the network such that it is unable to learn peculiarities. Early stopping
! aims at stopping the training at the point of optimal generalization by
! dividing the available data into training and validation sets. A description
! of the early stopping method can for instance be found in section 3.3 of 
! /pub/papers/techreports/1994-21.ps.gz on ftp.ira.uka.de (anonymous ftp). 
  
  ------------------------------------------------------------------------
  
! Subject: Why use a bias input? Why activation functions?
! ========================================================
  
  Consider a multilayer perceptron. Choose any hidden unit or output unit.
  Let's say there are N inputs to that unit, which define an N-dimensional
--- 131,427 ----
  method for computing the error gradient for a feedforward network, a
  straightforward but elegant application of the chain rule of elementary
! calculus (Werbos 1994). By extension, backpropagation or backprop refers
! to a training method that uses backpropagation to compute the gradient. By
! further extension, a backprop network is a feedforward network trained by
! backpropagation. 
! 
! Standard backprop is a euphemism for the generalized delta rule, the
! training algorithm that was popularized by Rumelhart, Hinton, and Williams
! in chapter 8 of Rumelhart and McClelland (1986), which remains the most
! widely used supervised training method for neural nets. The generalized
! delta rule (including momentum) is called the heavy ball method in the
! numerical analysis literature (Poljak 1964; Bertsekas 1995, 78-79). 
! 
! Standard backprop can be used for on-line training (in which the weights are
! updated after processing each case) but it does not converge. To obtain
! convergence, the learning rate must be slowly reduced. This methodology is
! called stochastic approximation. 
! 
! For batch processing, there is no reason to suffer through the slow
! convergence and the tedious tuning of learning rates and momenta of standard
! backprop. Much of the NN research literature is devoted to attempts to speed
! up backprop. Most of these methods are inconsequential; two that are
! effective are Quickprop (Fahlman 1989) and RPROP (Riedmiller and Braun
! 1993). But conventional methods for nonlinear optimization are usually
! faster and more reliable than any of the "props". See "What are conjugate
! gradients, Levenberg-Marquardt, etc.?". 
! 
! References on backprop: 
! 
!    Bertsekas, D. P. (1995), Nonlinear Programming, Belmont, MA: Athena
!    Scientific, ISBN 1-886529-14-0. 
! 
!    Poljak, B.T. (1964), "Some methods of speeding up the convergence of
!    iteration methods," Z. Vycisl. Mat. i Mat. Fiz., 4, 1-17. 
! 
!    Rumelhart, D.E., Hinton, G.E., and Williams, R.J. (1986), "Learning
!    internal representations by error propagation", in Rumelhart, D.E. and
!    McClelland, J. L., eds. (1986), Parallel Distributed Processing:
!    Explorations in the Microstructure of Cognition, Volume 1, 318-362,
!    Cambridge, MA: The MIT Press. 
! 
!    Werbos, P.J. (1994), The Roots of Backpropagation, NY: John Wiley &
!    Sons. 
! 
! References on stochastic approximation: 
! 
!    Robbins, H. & Monro, S. (1951), "A Stochastic Approximation Method",
!    Annals of Mathematical Statistics, 22, 400-407. 
! 
!    Kushner, H. & Clark, D. (1978), Stochastic Approximation Methods for
!    Constrained and Unconstrained Systems, Springer-Verlag. 
! 
!    White, H. (1989), "Some Asymptotic Results for Learning in Single Hidden
!    Layer Feedforward Network Models", J. of the American Statistical Assoc.,
!    84, 1008-1013. 
! 
! References on better props: 
! 
!    Fahlman, S.E. (1989), "Faster-Learning Variations on Back-Propagation: An
!    Empirical Study", in Touretzky, D., Hinton, G, and Sejnowski, T., eds., 
!    Proceedings of the 1988 Connectionist Models Summer School, Morgan
!    Kaufmann, 38-51. 
! 
!    Riedmiller, M. and Braun, H. (1993), "A Direct Adaptive Method for Faster
!    Backpropagation Learning: The RPROP Algorithm", Proceedings of the IEEE
!    International Conference on Neural Networks 1993, San Francisco: IEEE. 
  
  ------------------------------------------------------------------------
  
! Subject: What are conjugate gradients,
! ======================================
! Levenberg-Marquardt, etc.? 
! ===========================
! 
! Training a neural network is, in most cases, an exercise in numerical
! optimization of a usually nonlinear function. Methods of nonlinear
! optimization have been studied for hundreds of years, and there is a huge
! literature on the subject in fields such as numerical analysis, operations
! research, and statistical computing, e.g., Bertsekas 1995, Gill, Murray, and
! Wright 1981. There is no single best method for nonlinear optimization. You
! need to choose a method based on the characteristics of the problem to be
! solved. For functions with continuous second derivatives (which would
! include feedforward nets with the most popular differentiable activation
! functions and error functions), three general types of algorithms have been
! found to be effective for most practical purposes: 
! 
!  o For a small number of weights, stabilized Newton and Gauss-Newton
!    algorithms, including various Levenberg-Marquardt and trust-region
!    algorithms are efficient. 
!  o For a moderate number of weights, various quasi-Newton algorithms are
!    efficient. 
!  o For a large number of weights, various conjugate-gradient algorithms are
!    efficient. 
! 
! All of the above methods find local optima. For global optimization, there
! are a variety of approaches. You can simply run any of the local
! optimization methods from numerous random starting points. Or you can try
! more complicated methods designed for global optimization such as simulated
! annealing or genetic algorithms (see Reeves 1993 and "What about Genetic
! Algorithms and Evolutionary Computation?"). 
! 
! For a survey of optimization software, see More\' and Wright (1993). For
! more on-line information on numerical optimization see: 
! 
!  o The kangaroos, a nontechnical description of various optimization
!    methods, at ftp://ftp.sas.com/pub/neural/kangaroos. 
!  o John Gregory's nonlinear programming FAQ at 
!    http://www.skypoint.com/subscribers/ashbury/nonlinear-programming-faq.html.
!  o Arnold Neumaier's page on global optimization at 
!    http://solon.cma.univie.ac.at/~neum/glopt.html. 
! 
! References: 
! 
!    Bertsekas, D. P. (1995), Nonlinear Programming, Belmont, MA: Athena
!    Scientific, ISBN 1-886529-14-0. 
! 
!    Gill, P.E., Murray, W. and Wright, M.H. (1981) Practical Optimization,
!    Academic Press: London. 
! 
!    Levenberg, K. (1944) "A method for the solution of certain problems in
!    least squares," Quart. Appl. Math., 2, 164-168. 
! 
!    Marquardt, D. (1963) "An algorithm for least-squares estimation of
!    nonlinear parameters," SIAM J. Appl. Math., 11, 431-441. 
! 
!    More\', J.J. (1977) "The Levenberg-Marquardt algorithm: implementation
!    and theory," in Watson, G.A., ed., _Numerical Analysis_, Lecture Notes in
!    Mathematics 630, Springer-Verlag, Heidelberg, 105-116. 
! 
!    More\', J.J. and Wright, S.J. (1993), Optimization Software Guide,
!    Philadelphia: SIAM, ISBN 0-89871-322-6. 
! 
!    Reeves, C.R., ed. (1993) Modern Heuristic Techniques for Combinatorial
!    Problems, NY: Wiley. 
! 
!    Rinnooy Kan, A.H.G., and Timmer, G.T., (1989) Global Optimization: A
!    Survey, International Series of Numerical Mathematics, vol. 87, Basel:
!    Birkhauser Verlag. 
  
+ ------------------------------------------------------------------------
+ 
+ Subject: How should categories be coded? 
+ =========================================
+ 
+ First, consider unordered categories. If you want to classify cases into one
+ of C categories (i.e. you have a categorical target variable), use 1-of-C
+ coding. That means that you code C binary (0/1) target variables
+ corresponding to the C categories. Statisticians call these "dummy"
+ variables. Each dummy variable is given the value zero except for the one
+ corresponding to the correct category, which is given the value one. Then
+ use a softmax output activation function (see "What is a softmax activation
+ function?") so that the net, if properly trained, will produce valid
+ posterior probability estimates. If the categories are Red, Green, and Blue,
+ then the data would look like this: 
+ 
+    Category  Dummy variables
+    --------  ---------------
+     Red        1   0   0
+     Green      0   1   0
+     Blue       0   0   1
+ 
+ When there are only two categories, it is simpler to use just one dummy
+ variable with a logistic output activation function; this is equivalent to
+ using softmax with two dummy variables. 
+ 
+ The common practice of using target values of .1 and .9 instead of 0 and 1
+ prevents the outputs of the network from being directly interpretable as
+ posterior probabilities. 
+ 
+ Another common practice is to use a logistic activation function for each
+ output. Thus, the outputs are not constrained to sum to one, so they are not
+ valid posterior probability estimates. The usual justification advanced for
+ this procedure is that if a test case is not similar to any of the training
+ cases, all of the outputs will be small, indicating that the case cannot be
+ classified reliably. This claim is incorrect, since a test case that is not
+ similar to any of the training cases will require the net to extrapolate,
+ and extrapolation is thoroughly unreliable; such a test case may produce all
+ small outputs, all large outputs, or any combination of large and small
+ outputs. If you want a classification method that detects novel cases for
+ which the classification may not be reliable, you need a method based on
+ probability density estimation. For example, see "What is PNN?". 
+ 
+ It is very important not to use a single variable for an unordered
+ categorical target. Suppose you used a single variable with values 1, 2, and
+ 3 for red, green, and blue, and the training data with two inputs looked
+ like this: 
+ 
+       |    1    1
+       |   1   1
+       |       1   1
+       |     1   1
+       | 
+       |      X
+       | 
+       |    3   3           2   2
+       |     3     3      2
+       |  3   3            2    2
+       |     3   3       2    2
+       | 
+       +----------------------------
+ 
+ Consider a test point located at the X. The correct output would be that X
+ has about a 50-50 chance of being a 1 or a 3. But if you train with a single
+ target variable with values of 1, 2, and 3, the output for X will be the
+ average of 1 and 3, so the net will say that X is definitely a 2! 
+ 
+ For an input with categorical values, you can use 1-of-(C-1) coding if the
+ network has a bias unit. This is just like 1-of-C coding, except that you
+ omit one of the dummy variables (doesn't much matter which one). Using all C
+ of the dummy variables creates a linear dependency on the bias unit, which
+ is not advisable unless you are using weight decay or Bayesian estimation or
+ some such thing that requires all C weights to be treated on an equal basis.
+ 1-of-(C-1) coding looks like this: 
+ 
+    Category  Dummy variables
+    --------  ---------------
+     Red        1   0
+     Green      0   1
+     Blue       0   0
+ 
+ Another possible coding is called "effects" coding or "deviations from
+ means" coding in statistics. It is like 1-of-(C-1) coding, except that when
+ a case belongs to the category for the omitted dummy variable, all of the
+ dummy variables are set to -1, like this: 
+ 
+    Category  Dummy variables
+    --------  ---------------
+     Red        1   0
+     Green      0   1
+     Blue      -1  -1
+ 
+ As long as a bias unit is used, any network with effects coding can be
+ transformed into an equivalent network with 1-of-(C-1) coding by a linear
+ transformation of the weights. So the only advantage of effects coding is
+ that the dummy variables require no standardizing (see "Should I
+ normalize/standardize/rescale the data?"). 
+ 
+ Now consider ordered categories. For inputs, some people recommend a
+ "thermometer code" like this: 
+ 
+    Category  Dummy variables
+    --------  ---------------
+     Red        1   1   1
+     Green      0   1   1
+     Blue       0   0   1
+ 
+ However, thermometer coding is equivalent to 1-of-C coding, in that for any
+ network using 1-of-C coding, there exists a network with thermometer coding
+ that produces identical outputs; the weights in the thermometer-coded
+ network are just the differences of successive weights in the 1-of-C-coded
+ network. To get a genuinely ordinal representation, you must constrain the
+ weights connecting the dummy variables to the hidden units to be nonnegative
+ (except for the first dummy variable). 
+ 
+ It is often effective to represent an ordinal input as a single variable
+ like this: 
+ 
+    Category  Input
+    --------  -----
+     Red        1
+     Green      2
+     Blue       3
+ 
+ Although this representation involves only a single quantitative input,
+ given enough hidden units, the net is capable of computing nonlinear
+ transformations of that input that will produce results equivalent to any of
+ the dummy coding schemes. But using a single quantitative input makes it
+ easier for the net to use the order of the categories to generalize when

==> nn3.changes.body <==
*** nn3.oldbody.Wed Feb 28 23:00:28 1996
--- nn3.body.Thu Mar 28 23:00:17 1996
***************
*** 1,4 ****
  Archive-name: ai-faq/neural-nets/part3
! Last-modified: 1996-01-27
  URL: ftp://ftp.sas.com/pub/neural/FAQ3.html
  Maintainer: saswss@unx.sas.com (Warren S. Sarle)
--- 1,4 ----
  Archive-name: ai-faq/neural-nets/part3
! Last-modified: 1996-03-14
  URL: ftp://ftp.sas.com/pub/neural/FAQ3.html
  Maintainer: saswss@unx.sas.com (Warren S. Sarle)
***************
*** 12,34 ****
  
  Part 1: Introduction
- 
-    What is this newsgroup for? How shall it be used?
-    What is a neural network (NN)?
-    What can you do with a Neural Network and what not?
-    Who is concerned with Neural Networks?
- 
  Part 2: Learning
- 
-    What does 'backprop' mean? What is 'overfitting'?
-    Why use a bias input? Why activation functions?
-    How many hidden units should I use?
-    How many learning methods for NNs exist? Which?
-    What about Genetic Algorithms and Evolutionary Computation?
-    What about Fuzzy Logic?
-    How are NNs related to statistical methods?
- 
  Part 3: Information resources
  
!    Good introductory literature about Neural Networks?
     Any journals and magazines about Neural Networks?
     The most important conferences concerned with Neural Networks?
--- 12,19 ----
  
  Part 1: Introduction
  Part 2: Learning
  Part 3: Information resources
  
!    Good literature about Neural Networks?
     Any journals and magazines about Neural Networks?
     The most important conferences concerned with Neural Networks?
***************
*** 37,67 ****
  
  Part 4: Datasets
- 
-    Databases for experimentation with NNs?
- 
  Part 5: Free software
  
!    Freely available software packages for NN simulation?
  
! Part 6: Commercial software
  
!    Commercial software packages for NN simulation?
  
! Part 7: Hardware
  
!    Neural Network hardware?
  
! ------------------------------------------------------------------------
  
! Subject: Good introductory literature about Neural
! ==================================================
! Networks?
! =========
  
! 0.) The best (you can flame me if you do it entertainingly):
! ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  
! Bishop, C.M. (1995). Neural Networks for Pattern Recognition, Oxford: Oxford
! University Press. ISBN 0-19-853849-9 (hardback) or 0-19-853864-2
  (paperback), xvii+482 pages.
  This is definitely the best book on neural nets for practical applications
--- 22,58 ----
  
  Part 4: Datasets
  Part 5: Free software
+ Part 6: Commercial software
+ Part 7: Hardware
  
! ------------------------------------------------------------------------
  
! Subject: Good literature about Neural Networks?
! ===============================================
  
! The Best
! ++++++++
  
! The best popular introduction to NNs
! ------------------------------------
  
! Hinton, G.E. (1992), "How Neural Networks Learn from Experience", Scientific
! American, 267 (September), 144-151. 
  
! The best elementary textbooks on NNs
! ------------------------------------
  
! Masters, Timothy (1994). Practical Neural Network Recipes in C++, Academic
! Press, ISBN 0-12-479040-2, US $45 incl. disks.
! "Lots of very good practical advice which most other books lack."
! 
! Weiss, S.M. & Kulikowski, C.A. (1991), Computer Systems That Learn,
! Morgan Kaufmann. ISBN 1 55860 065 5. 
  
! The best intermediate textbooks on NNs
! --------------------------------------
  
! Bishop, C.M. (1995). Neural Networks for Pattern Recognition, Oxford:
! Oxford University Press. ISBN 0-19-853849-9 (hardback) or 0-19-853864-2
  (paperback), xvii+482 pages.
  This is definitely the best book on neural nets for practical applications
***************
*** 97,103 ****
  to get through."
  
! Masters, Timothy (1994). Practical Neural Network Recipes in C++. Academic
! Press, ISBN 0-12-479040-2, US $45 incl. disks.
! "Lots of very good practical advice which most other books lack."
  
  Ripley, B.D. (1996) Pattern Recognition and Neural Networks, Cambridge:
--- 88,93 ----
  to get through."
  
! The best advanced textbook covering NNs
! ---------------------------------------
  
  Ripley, B.D. (1996) Pattern Recognition and Neural Networks, Cambridge:
***************
*** 112,118 ****
  examples in his book will be of interest even to beginners in neural nets.
  
! 1.) Books for the beginner:
! +++++++++++++++++++++++++++
  
  Aleksander, I. and Morton, H. (1990). An Introduction to Neural Computing.
  Chapman and Hall. (ISBN 0-412-37780-2). Comments: "This book seems to be
--- 102,138 ----
  examples in his book will be of interest even to beginners in neural nets.
  
! The best books on image and signal processing with NNs
! ------------------------------------------------------
! 
! Masters, T. (1994), Signal and Image Processing with Neural Networks: A
! C++ Sourcebook, Wiley.
! 
! Cichocki, A. and Unbehauen, R. (1993). Neural Networks for Optimization
! and Signal Processing. NY: John Wiley & Sons, ISBN 0-471-930105 (hardbound),
! 526 pages, $57.95. 
! "Partly a textbook and partly a research monograph; introduces the basic
! concepts, techniques, and models related to neural networks and
! optimization, excluding rigorous mathematical details. Accessible to a wide
! readership with a differential calculus background. The main coverage of the
! book is on recurrent neural networks with continuous state variables. The
! book title would be more appropriate without mentioning signal processing.
! Well edited, good illustrations."
! 
! The best book on time-series forecasting with NNs
! -------------------------------------------------
! 
! Weigend, A.S. and Gershenfeld, N.A., eds. (1994) Time Series Prediction:
! Forecasting the Future and Understanding the Past, Addison-Wesley: Reading,
! MA. 
! 
! The best comparison of NNs with other classification methods
! ------------------------------------------------------------
  
+ Michie, D., Spiegelhalter, D.J. and Taylor, C.C. (1994), Machine Learning,
+ Neural and Statistical Classification, Ellis Horwood. 
+ 
+ Books for the Beginner:
+ +++++++++++++++++++++++
+ 
  Aleksander, I. and Morton, H. (1990). An Introduction to Neural Computing.
  Chapman and Hall. (ISBN 0-412-37780-2). Comments: "This book seems to be
***************
*** 145,150 ****
  World Wide Web.
  
! Haykin, S. (1994). Neural Networks, a Comprehensive Foundation. Macmillan,
! New York, NY.
  "A very readable, well written intermediate text on NNs Perspective is
  primarily one of pattern recognition, estimation and signal processing.
--- 165,170 ----
  World Wide Web.
  
! Haykin, S. (1994). Neural Networks, a Comprehensive Foundation.
! Macmillan, New York, NY.
  "A very readable, well written intermediate text on NNs Perspective is
  primarily one of pattern recognition, estimation and signal processing.
***************
*** 156,161 ****
  1000 references."
  
! Hecht-Nielsen, R. (1990). Neurocomputing. Addison Wesley. Comments: "A good
! book", "comprises a nice historical overview and a chapter about NN
  hardware. Well structured prose. Makes important concepts clear."
  
--- 176,181 ----
  1000 references."
  
! Hecht-Nielsen, R. (1990). Neurocomputing. Addison Wesley. Comments: "A
! good book", "comprises a nice historical overview and a chapter about NN
  hardware. Well structured prose. Makes important concepts clear."
  
***************
*** 196,202 ****
  accompanies a software package, but I haven't seen that yet".
  
! Rao, V.B & H.V. (1993). C++ Neural Networks and Fuzzy Logic. MIS:Press, ISBN
! 1-55828-298-x, US $45 incl. disks. "Probably not 'leading edge' stuff but
! detailed enough to get your hands dirty!"
  
  Wasserman, P. D. (1989). Neural Computing: Theory & Practice. Van Nostrand
--- 216,222 ----
  accompanies a software package, but I haven't seen that yet".
  
! Rao, V.B & H.V. (1993). C++ Neural Networks and Fuzzy Logic. MIS:Press,
! ISBN 1-55828-298-x, US $45 incl. disks. "Probably not 'leading edge' stuff
! but detailed enough to get your hands dirty!"
  
  Wasserman, P. D. (1989). Neural Computing: Theory & Practice. Van Nostrand
***************
*** 213,225 ****
  anything. An excellent book to give your manager."
  
! Wasserman, P.D. (1993). Advanced Methods in Neural Computing. Van Nostrand
! Reinhold: New York (ISBN: 0-442-00461-3). Comments: Several neural network
! topics are discussed e.g. Probalistic Neural Networks, Backpropagation and
! beyond, neural control, Radial Basis Function Networks, Neural Engineering.
! Furthermore, several subjects related to neural networks are mentioned e.g.
! genetic algorithms, fuzzy logic, chaos. Just the functionality of these
! subjects is described; enough to get you started. Lots of references are
! given to more elaborate descriptions. Easy to read, no extensive
! mathematical background necessary.
  
  Zurada, Jacek M. (1992). Introduction To Artificial Neural Systems.
--- 233,245 ----
  anything. An excellent book to give your manager."
  
! Wasserman, P.D. (1993). Advanced Methods in Neural Computing. Van
! Nostrand Reinhold: New York (ISBN: 0-442-00461-3). Comments: Several neural
! network topics are discussed e.g. Probalistic Neural Networks,
! Backpropagation and beyond, neural control, Radial Basis Function Networks,
! Neural Engineering. Furthermore, several subjects related to neural networks
! are mentioned e.g. genetic algorithms, fuzzy logic, chaos. Just the
! functionality of these subjects is described; enough to get you started.
! Lots of references are given to more elaborate descriptions. Easy to read,
! no extensive mathematical background necessary.
  
  Zurada, Jacek M. (1992). Introduction To Artificial Neural Systems.
***************
*** 237,242 ****
  Applications, Implementations, Appendix) 
  
! 2.) The classics:
! +++++++++++++++++
  
  Kohonen, T. (1984). Self-organization and Associative Memory.
--- 257,262 ----
  Applications, Implementations, Appendix) 
  
! The Classics:
! +++++++++++++
  
  Kohonen, T. (1984). Self-organization and Associative Memory.
***************
*** 251,256 ****
  $65 for both volumes)."; "THE Connectionist bible".
  
! 3.) Introductory journal articles:
! ++++++++++++++++++++++++++++++++++
  
  Hinton, G. E. (1989). Connectionist learning procedures. Artificial
--- 271,276 ----
  $65 for both volumes)."; "THE Connectionist bible".
  
! Introductory Journal Articles:
! ++++++++++++++++++++++++++++++
  
  Hinton, G. E. (1989). Connectionist learning procedures. Artificial
***************
*** 267,279 ****
  vol. 1, no. 1. pp. 3-16. Comments: "A general review".
  
! 4.) Not-quite-so-introductory literature:
! +++++++++++++++++++++++++++++++++++++++++
  
! Anderson, J. A. and Rosenfeld, E. (Eds). (1988). Neurocomputing: Foundations
! of Research. The MIT Press: Cambridge, MA. Comments: "An expensive book, but
! excellent for reference. It is a collection of reprints of most of the major
! papers in the field." 
  
! Anderson, J. A., Pellionisz, A. and Rosenfeld, E. (Eds). (1990).
  Neurocomputing 2: Directions for Research. The MIT Press: Cambridge, MA.
  Comments: "The sequel to their well-known Neurocomputing book."
--- 287,299 ----
  vol. 1, no. 1. pp. 3-16. Comments: "A general review".
  
! Not-quite-so-introductory Literature:
! +++++++++++++++++++++++++++++++++++++
  
! Anderson, J. A. and Rosenfeld, E. (Eds). (1988). Neurocomputing:
! Foundations of Research. The MIT Press: Cambridge, MA. Comments: "An
! expensive book, but excellent for reference. It is a collection of reprints
! of most of the major papers in the field." 
  
! Anderson, J. A., Pellionisz, A. and Rosenfeld, E. (Eds). (1990). 
  Neurocomputing 2: Directions for Research. The MIT Press: Cambridge, MA.
  Comments: "The sequel to their well-known Neurocomputing book."
***************
*** 284,305 ****
  research in the area".
  
! Cichocki, A. and Unbehauen, R. (1994). Neural Networks for Optimization and
! Signal Processing. John Wiley & Sons, West Sussex, England, 1993, ISBN
! 0-471-930105 (hardbound), 526 pages, $57.95. "Partly a textbook and partly a
! research monograph; introduces the basic concepts, techniques, and models
! related to neural networks and optimization, excluding rigorous mathematical
! details. Accessible to a wide readership with a differential calculus
! background. The main coverage of the book is on recurrent neural networks
! with continuous state variables. The book title would be more appropriate
! without mentioning signal processing. Well edited, good illustrations."
! 
! Khanna, T. (1990). Foundations of Neural Networks. Addison-Wesley: New York.
! Comments: "Not so bad (with a page of erroneous formulas (if I remember
! well), and #hidden layers isn't well described)."; "Khanna's intention in
! writing his book with math analysis should be commended but he made several
! mistakes in the math part".
  
! Kung, S.Y. (1993). Digital Neural Networks, Prentice Hall, Englewood Cliffs,
! NJ.
  
  Levine, D. S. (1990). Introduction to Neural and Cognitive Modeling.
--- 304,315 ----
  research in the area".
  
! Khanna, T. (1990). Foundations of Neural Networks. Addison-Wesley: New
! York. Comments: "Not so bad (with a page of erroneous formulas (if I
! remember well), and #hidden layers isn't well described)."; "Khanna's
! intention in writing his book with math analysis should be commended but he
! made several mistakes in the math part".
  
! Kung, S.Y. (1993). Digital Neural Networks, Prentice Hall, Englewood
! Cliffs, NJ.
  
  Levine, D. S. (1990). Introduction to Neural and Cognitive Modeling.
***************
*** 345,348 ****
--- 355,402 ----
  employs a strong interdisciplinary emphasis".
  
+ The Worst
+ +++++++++
+ 
+    Blum, Adam (1992), Neural Networks in C++, Wiley. 
+ 
+    Welstead, Stephen T. (1994), Neural Network and Fuzzy Logic
+    Applications in C/C++, Wiley. 
+ 
+ Both Blum and Welstead contribute to the dangerous myth that any idiot can
+ use a neural net by dumping in whatever data are handy and letting it train
+ for a few days. They both have little or no discussion of generalization,
+ validation, and overfitting. Neither provides any valid advice on choosing
+ the number of hidden nodes. If you have ever wondered where these stupid
+ "rules of thumb" that pop up frequently come from, here's a source for one
+ of them: 
+ 
+    "A rule of thumb is for the size of this [hidden] layer to be
+    somewhere between the input layer size ... and the output layer size
+    ..." Blum, p. 60. 
+ 
+ (John Lazzaro tells me he recently "reviewed a paper that cited this rule of
+ thumb--and referenced this book! Needless to say, the final version of that
+ paper didn't include the reference!") 
+ 
+ Blum offers some profound advice on choosing inputs: 
+ 
+    "The next step is to pick as many input factors as possible that
+    might be related to [the target]." 
+ 
+ Blum also shows a deep understanding of statistics: 
+ 
+    "A statistical model is simply a more indirect way of learning
+    correlations. With a neural net approach, we model the problem
+    directly." p. 8. 
+ 
+ Blum at least mentions some important issues, however simplistic his advice
+ may be. Welstead just ignores them. What Welstead gives you is code--vast
+ amounts of code. I have no idea how anyone could write that much code for a
+ simple feedforward NN. Welstead's approach to validation, in his chapter on
+ financial forecasting, is to reserve two cases for the validation set! 
+ 
+ My comments apply only to the text of the above books. I have not examined
+ or attempted to compile the code. 
+ 
  ------------------------------------------------------------------------
  
***************
*** 763,767 ****
     "Upcoming Neural Network Conferences", which lists names, dates,
     locations, contacts, and deadlines. It is also available on the WWW from 
!    http://www.neuronet.ph.kcl.ac.uk/neuronet/bakker.html 
  
  ------------------------------------------------------------------------
--- 817,823 ----
     "Upcoming Neural Network Conferences", which lists names, dates,
     locations, contacts, and deadlines. It is also available on the WWW from 
!    http://www.neuronet.ph.kcl.ac.uk/neuronet/bakker.html. 
! 3. The IEEE Neural Network Council maintains an up-to-date list of
!    conferences at http://www.ieee.org/nnc. 
  
  ------------------------------------------------------------------------
***************
*** 887,892 ****
     EchoMail compatible bulletin board systems. 
  
! 5. Neural ftp archive site ftp.funet.fi
! +++++++++++++++++++++++++++++++++++++++
  
     Is administrating a large collection of neural network papers and
--- 943,955 ----
     EchoMail compatible bulletin board systems. 
  
! 5. Neuroprose ftp archive site: archive.cis.ohio-state.edu
! ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
! 
!    ftp://archive.cis.ohio-state.edu/pub/neuroprose
!    This directory contains technical reports as a public service to the
!    connectionist and neural network scientific community. 
! 
! 6. Neural ftp archive site: ftp.funet.fi
! ++++++++++++++++++++++++++++++++++++++++
  
     Is administrating a large collection of neural network papers and
***************
*** 897,908 ****
     at fastest. Contact: neural-adm@ftp.funet.fi 
  
! 6. USENET newsgroup comp.org.issnnet
  ++++++++++++++++++++++++++++++++++++
  
     Forum for discussion of academic/student-related issues in NNs, as well
!    as information on ISSNNet (see question 'associations') and its
     activities. 
  
! 7. AI CD-ROM
  ++++++++++++
  
--- 960,971 ----
     at fastest. Contact: neural-adm@ftp.funet.fi 
  
! 7. USENET newsgroup comp.org.issnnet
  ++++++++++++++++++++++++++++++++++++
  
     Forum for discussion of academic/student-related issues in NNs, as well
!    as information on ISSNNet (see question "associations") and its
     activities. 
  
! 8. AI CD-ROM
  ++++++++++++
  
***************
*** 929,933 ****
     details) 
  
! 8. NN events server
  +++++++++++++++++++
  
--- 992,996 ----
     details) 
  
! 9. NN events server
  +++++++++++++++++++
  
***************
*** 936,941 ****
     http://www.idiap.ch/html/idiap-networks.html. 
  
! 9. World Wide Web
! +++++++++++++++++
  
     In World-Wide-Web (WWW, for example via the xmosaic program) you can read
--- 999,1004 ----
     http://www.idiap.ch/html/idiap-networks.html. 
  
! 10. World Wide Web
! ++++++++++++++++++
  
     In World-Wide-Web (WWW, for example via the xmosaic program) you can read
***************
*** 952,956 ****
     Many others are available too; WWW is changing all the time. 
  

==> nn4.changes.body <==
*** nn4.oldbody.Wed Feb 28 23:00:31 1996
--- nn4.body.Thu Mar 28 23:00:20 1996
***************
*** 12,39 ****
  
  Part 1: Introduction
- 
-    What is this newsgroup for? How shall it be used?
-    What is a neural network (NN)?
-    What can you do with a Neural Network and what not?
-    Who is concerned with Neural Networks?
- 
  Part 2: Learning
- 
-    What does 'backprop' mean? What is 'overfitting'?
-    Why use a bias input? Why activation functions?
-    How many hidden units should I use?
-    How many learning methods for NNs exist? Which?
-    What about Genetic Algorithms and Evolutionary Computation?
-    What about Fuzzy Logic?
-    How are NNs related to statistical methods?
- 
  Part 3: Information resources
- 
-    Good introductory literature about Neural Networks?
-    Any journals and magazines about Neural Networks?
-    The most important conferences concerned with Neural Networks?
-    Neural Network Associations?
-    Other sources of information about NNs?
- 
  Part 4: Datasets
  
--- 12,17 ----
***************
*** 41,55 ****
  
  Part 5: Free software
- 
-    Freely available software packages for NN simulation?
- 
  Part 6: Commercial software
- 
-    Commercial software packages for NN simulation?
- 
  Part 7: Hardware
  
-    Neural Network hardware?
- 
  ------------------------------------------------------------------------
  
--- 19,25 ----
***************
*** 200,204 ****
     hull@cs.buffalo.edu (email) 
  
! 6. AI-CD-ROM (see question 'Other sources of information')
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  
--- 170,174 ----
     hull@cs.buffalo.edu (email) 
  
! 6. AI-CD-ROM (see question "Other sources of information")
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  

==> nn5.changes.body <==
*** nn5.oldbody.Wed Feb 28 23:00:35 1996
--- nn5.body.Thu Mar 28 23:00:27 1996
***************
*** 1,4 ****
  Archive-name: ai-faq/neural-nets/part5
! Last-modified: 1996-01-17
  URL: ftp://ftp.sas.com/pub/neural/FAQ5.html
  Maintainer: saswss@unx.sas.com (Warren S. Sarle)
--- 1,4 ----
  Archive-name: ai-faq/neural-nets/part5
! Last-modified: 1996-03-06
  URL: ftp://ftp.sas.com/pub/neural/FAQ5.html
  Maintainer: saswss@unx.sas.com (Warren S. Sarle)
***************
*** 12,43 ****
  
  Part 1: Introduction
- 
-    What is this newsgroup for? How shall it be used?
-    What is a neural network (NN)?
-    What can you do with a Neural Network and what not?
-    Who is concerned with Neural Networks?
- 
  Part 2: Learning
- 
-    What does 'backprop' mean? What is 'overfitting'?
-    Why use a bias input? Why activation functions?
-    How many hidden units should I use?
-    How many learning methods for NNs exist? Which?
-    What about Genetic Algorithms and Evolutionary Computation?
-    What about Fuzzy Logic?
-    How are NNs related to statistical methods?
- 
  Part 3: Information resources
- 
-    Good introductory literature about Neural Networks?
-    Any journals and magazines about Neural Networks?
-    The most important conferences concerned with Neural Networks?
-    Neural Network Associations?
-    Other sources of information about NNs?
- 
  Part 4: Datasets
- 
-    Databases for experimentation with NNs?
- 
  Part 5: Free software
  
--- 12,18 ----
***************
*** 45,55 ****
  
  Part 6: Commercial software
- 
-    Commercial software packages for NN simulation?
- 
  Part 7: Hardware
  
-    Neural Network hardware?
- 
  ------------------------------------------------------------------------
  
--- 20,25 ----
***************
*** 412,416 ****
  
     This is just a small example program. Available for anonymous ftp from
!    park.bu.edu [128.176.121.56] /pub/fuzzy-artmap.tar.Z (44 kB). 
  
  22. PYGMALION
--- 382,387 ----
  
     This is just a small example program. Available for anonymous ftp from
!    park.bu.edu [128.176.121.56] ftp://cns-ftp.bu.edu/pub/fuzzy-artmap.tar.Z
!    (44 kB). 
  
  22. PYGMALION
***************
*** 601,605 ****
  If you are using a small computer (PC, Mac, etc.) you may want to have a
  look at the Central Neural System Electronic Bulletin Board (see question 
! 'Other sources of information'). Modem: 409-737-5312; Sysop: Wesley R.
  Elsberry; 4160 Pirates' Beach, Galveston, TX, USA; welsberr@orca.tamu.edu.
  There are lots of small simulator packages, the CNS ANNSIM file set. There
--- 572,576 ----
  If you are using a small computer (PC, Mac, etc.) you may want to have a
  look at the Central Neural System Electronic Bulletin Board (see question 
! "Other sources of information"). Modem: 409-737-5312; Sysop: Wesley R.
  Elsberry; 4160 Pirates' Beach, Galveston, TX, USA; welsberr@orca.tamu.edu.
  There are lots of small simulator packages, the CNS ANNSIM file set. There
***************
*** 610,613 ****
  ------------------------------------------------------------------------
  
! Next part is part 6 (of 7). Previous part is part 4. 
  
--- 581,584 ----
  ------------------------------------------------------------------------
  
! Next part is part 6 (of 7). Previous part is part 4. @
  

==> nn6.changes.body <==
*** nn6.oldbody.Wed Feb 28 23:00:40 1996
--- nn6.body.Thu Mar 28 23:00:30 1996
***************
*** 1,4 ****
  Archive-name: ai-faq/neural-nets/part6
! Last-modified: 1996-02-22
  URL: ftp://ftp.sas.com/pub/neural/FAQ6.html
  Maintainer: saswss@unx.sas.com (Warren S. Sarle)
--- 1,4 ----
  Archive-name: ai-faq/neural-nets/part6
! Last-modified: 1996-03-28
  URL: ftp://ftp.sas.com/pub/neural/FAQ6.html
  Maintainer: saswss@unx.sas.com (Warren S. Sarle)
***************
*** 12,47 ****
  
  Part 1: Introduction
- 
-    What is this newsgroup for? How shall it be used?
-    What is a neural network (NN)?
-    What can you do with a Neural Network and what not?
-    Who is concerned with Neural Networks?
- 
  Part 2: Learning
- 
-    What does 'backprop' mean? What is 'overfitting'?
-    Why use a bias input? Why activation functions?
-    How many hidden units should I use?
-    How many learning methods for NNs exist? Which?
-    What about Genetic Algorithms and Evolutionary Computation?
-    What about Fuzzy Logic?
-    How are NNs related to statistical methods?
- 
  Part 3: Information resources
- 
-    Good introductory literature about Neural Networks?
-    Any journals and magazines about Neural Networks?
-    The most important conferences concerned with Neural Networks?
-    Neural Network Associations?
-    Other sources of information about NNs?
- 
  Part 4: Datasets
- 
-    Databases for experimentation with NNs?
- 
  Part 5: Free software
- 
-    Freely available software packages for NN simulation?
- 
  Part 6: Commercial software
  
--- 12,19 ----
***************
*** 50,55 ****
  Part 7: Hardware
  
-    Neural Network hardware?
- 
  ------------------------------------------------------------------------
  
--- 22,25 ----
***************
*** 73,77 ****
  1. nn/xnn 
  2. BrainMaker 
! 3. SAS Macros for Neural Networks 
  4. NeuralWorks 
  5. MATLAB Neural Network Toolbox 
--- 43,47 ----
  1. nn/xnn 
  2. BrainMaker 
! 3. SAS Software for Neural Networks 
  4. NeuralWorks 
  5. MATLAB Neural Network Toolbox 
***************
*** 192,196 ****
        The book Introduction to Neural Networks
        BrainMaker Users Guide and reference manual
!           300 pages , fully indexed, with tutorials, and sample networks
        Netmaker
            Netmaker makes building and training Neural Networks easy, by
--- 162,166 ----
        The book Introduction to Neural Networks
        BrainMaker Users Guide and reference manual
!           300 pages, fully indexed, with tutorials, and sample networks
        Netmaker
            Netmaker makes building and training Neural Networks easy, by
***************
*** 223,262 ****
       Introduction to Neural Networks 324 pp book
  
! 3. SAS Macros for Neural Networks
! +++++++++++++++++++++++++++++++++
  
!           Name: SAS Software
!        Company: SAS Institute, Inc.
!        Address: SAS Campus Drive, Cary, NC 27513, USA
!          Phone: (919) 677-8000
!          Email: saswss@unx.sas.com (Neural net inquiries only)
!            URL: ftp://ftp.sas.com/pub/neural/README
!    Operating system: Lots
!    System requirements: Lots
! 
!    Several SAS macros for feedforward neural nets are available
!    for releases 6.08 and later. For a list of macros and
!    articles relating to neural networks, see ftp://ftp.sas.com/pub/neural/README.
!    The macros are free but won't do you any good unless you
!    have licensed the required SAS products. If you want
!    information about licensing SAS products, call 919 677-8000
!    and ask for Software Sales.
! 
!    TNN is an elaborate system of macros for feedforward neural
!    nets including a variety of built-in activation and error
!    functions, multiple hidden layers, direct input-output
!    connections, missing value handling, categorical variables,
!    standardization of inputs and targets, and multiple
!    preliminary optimizations from random initial values to
!    avoid local minima.  TNN requires the SAS/OR product in
!    release 6.08 or later.  Release 6.10 or later is strongly
!    recommended. Release 6.10 is required for the plotting
!    macros to use SAS/INSIGHT.
! 
!    NETIML is a collection of SAS/IML modules and macros for
!    training and running multilayer perceptrons with a variety
!    of activation and error functions. NETIML requires the
!    SAS/IML product in release 6.08 or later.
  
  
  4. NeuralWorks
--- 193,249 ----
       Introduction to Neural Networks 324 pp book
  
! 3. SAS Software for Neural Networks
! +++++++++++++++++++++++++++++++++++
  
!        Name: SAS Software
  
+              In USA:                 In Europe:
+     Company: SAS Institute, Inc.     SAS Institute, European Office 
+     Address: SAS Campus Drive        Neuenheimer Landstrasse 28-30 
+              Cary, NC 27513          P.O.Box 10 53 40 
+              USA                     D-69043 Heidelberg 
+                                      Germany
+       Phone: (919) 677-8000          (49) 6221 4160
+         Fax: (919) 677-4444          (49) 6221 474 850
+       Email: saswss@unx.sas.com (Neural net macros)
+              eurgxh@mvs.sas.com (Neural net GUI)
+         URL: ftp://ftp.sas.com/pub/neural/README
+    Operating systems for macros: MS Windows (3.1, 95, NT) IBM OS/2 (2.1, 3.0, Warp),
+       MVS, VM/CMS, VSE/ESA, OpenVMS, ULTRIX, Digital UNIX, DG-UX, HP/UX,
+       Solaris, AIX, ConvexOS, MIPS ABI, INTEL ABI, Novell UNIXware,
+       Macintosh System 7.5, PowerPC 
+    Operating systems for GUI: Windows 3.1, OS/2, HP/UX, Solaris, AIX
+    System requirements: Lots of memory and disk space, floating point hardware
+    Comments: Oriented toward data analysis and statistical applications
+ 
+    Several SAS macros for feedforward neural nets are available for release
+    6.08 and later. For a list of macros and articles relating to neural
+    networks, see ftp://ftp.sas.com/pub/neural/README. The macros are free
+    but won't do you any good unless you have licensed the required SAS
+    products. If you want information about licensing SAS products, call one
+    of the phone numbers listed above and ask for Software Sales. 
+ 
+    There is also the SAS Neural Network Application including a graphical
+    user interface, on-site training and customisation. For prices and other
+    information, send email to eurgxh@mvs.sas.com or call the European
+    office. 
+ 
+    TNN is an elaborate system of macros for feedforward neural nets
+    including multilayer perceptrons, radial basis functions, statistical
+    versions of counterpropagation and learning vector quantization, a
+    variety of built-in activation and error functions, multiple hidden
+    layers, direct input-output connections, missing value handling,
+    categorical variables, standardization of inputs and targets, and
+    multiple preliminary optimizations from random initial values to avoid
+    local minima. Training is done by state-of-the-art numerical optimization
+    algorithms instead of tedious backprop. Maximum likelihood and
+    hierarchical Bayesian training are provided for a wide range of noise
+    distributions. TNN requires the SAS/OR product in release 6.08 or later.
+    Release 6.10 or later is strongly recommended. Release 6.10 is required
+    for the plotting macros to use SAS/INSIGHT. 
+ 
+    NETIML is a collection of SAS/IML modules and macros for training and
+    running multilayer perceptrons with a variety of activation and error
+    functions. NETIML requires the SAS/IML product in release 6.08 or later. 
  
  4. NeuralWorks
***************
*** 325,338 ****
     functions to address a specific application.
  
-    (Comment by Richard Andrew Miles Outerbridge, RAMO@UVPHYS.PHYS.UVIC.CA):
-    Matlab is spreading like hotcakes (and the educational discounts are very
-    impressive). The newest release of Matlab (4.0) answers the question "if
-    you could only program in one language what would it be?". The neural
-    network toolkit is worth getting for the manual alone. Matlab is
-    available with lots of other toolkits (signal processing, optimization,
-    etc.) but I don't use them much - the main package is more than enough.
-    The nice thing about the Matlab approach is that you can easily interface
-    the neural network stuff with anything else you are doing. 
- 
     (Comment from Nigel Dodd, nd@neural.win-uk.net): there is also a free
     Neural Network Based System Identification Toolbox available from 
--- 312,315 ----
***************
*** 342,345 ****
--- 319,329 ----
     pruning which the costly one doesn't attempt (as of Nov 1995). 
  
+    (Message from Eric A. Wan, ericwan@eeap.ogi.edu) FIR/TDNN Toolbox for
+    MATLAB: Beta version of a toolbox for FIR (Finite Impulse Response) and
+    TD (Time Delay) Neural Networks. For efficient stochastic implementation,
+    algorithms are written as MEX compatible c-code which can be called as
+    primitive functions from within MATLAB. Both source and compiled
+    functions are available. URL: http://www.eeap.ogi.edu/~ericwan/fir.html
+ 
  6. Propagator
  +++++++++++++
***************
*** 373,380 ****
     accel@technet.sg, accel@singapore.com
  
-    Readers' Choice 1994 - Technical Analysis of Stocks and Commodities.
-    Outperforms the others........ Technical Analysis of Stocks and
-    Commodities, May 95 
- 
     Neuroforecaster 4.1a for Windows is priced at US$1199 per single user
     license. Please email us (accel@technet.sg) for order form. 
--- 357,360 ----
***************
*** 430,454 ****
  +++++++++++++++++++++++++++
  
!    530 Fifth Avenue; New York, NY 10036; USA; Tel.: 001-212-398-7955
  
-    Founders: Dr. Leon Cooper (having a Nobel Price) and Dr. Charles Elbaum
-    (Brown University). Neural Network Models: Adaptive shape and pattern
-    recognition (Restricted Coulomb Energy - RCE) developed by NESTOR is one
-    of the most powerfull Neural Network Model used in a later products. The
-    basis for NESTOR products is the Nestor Learning System - NLS. Later are
-    developed: Character Learning System - CLS and Image Learning System -
-    ILS. Nestor Development System - NDS is a development tool in Standard C
-    - one of the most powerfull PC-Tools for simulation and development of
-    Neural Networks. NLS is a multi-layer, feed forward system with low
-    connectivity within each layer and no relaxation procedure used for
-    determining an output response. This unique architecture allows the NLS
-    to operate in real time without the need for special computers or custom
-    hardware. NLS is composed of multiple neural networks, each specializing
-    in a subset of information about the input patterns. The NLS integrates
-    the responses of its several parallel networks to produce a system
-    response that is far superior to that of other neural networks. Minimized
-    connectivity within each layer results in rapid training and efficient
-    memory utilization- ideal for current VLSI technology. Intel has made
-    such a chip - NE1000. 
  
  9. NeuroShell2/NeuroWindows
--- 410,445 ----
  +++++++++++++++++++++++++++
  
!    530 Fifth Avenue;
!    New York, NY 10036; USA;
!    Tel.: 001-212-398-7955
! 
!    Founders:
!    Dr. Leon Cooper (having a Nobel Price) and Dr. Charles Elbaum
!    (Brown University).
! 
!    Neural Network Models:
!    Adaptive shape and pattern recognition (Restricted Coulomb Energy - RCE)
!    developed by NESTOR is one of the most powerfull Neural Network Model
!    used in a later products.
! 
!    The basis for NESTOR products is the Nestor Learning System - NLS. Later
!    are developed: Character Learning System - CLS and Image Learning System
!    - ILS.  Nestor Development System - NDS is a development tool in
!    Standard C - a powerfull PC-Tool for simulation and development of
!    Neural Networks.
! 
!    NLS is a multi-layer, feed forward system with low connectivity within
!    each layer and no relaxation procedure used for determining an output
!    response.  This unique architecture allows the NLS to operate in real
!    time without the need for special computers or custom hardware.
! 
!    NLS is composed of multiple neural networks, each specializing in a
!    subset of information about the input patterns. The NLS integrates the
!    responses of its several parallel networks to produce a system response.
! 
!    Minimized connectivity within each layer results in rapid training and
!    efficient memory utilization- ideal for current VLSI technology. Intel
!    has made such a chip - NE1000.
  
  
  9. NeuroShell2/NeuroWindows
***************
*** 475,480 ****
     price: $369.
  
!    Contact: Ward Systems Group, Inc.; Executive Park West; 5 Hillcrest
!    Drive; Frederick, MD 21702; USA; Phone: 301 662-7950; FAX: 301 662-5666.
     Contact us for a free demo diskette and Consumer's Guide to Neural
     Networks. 
--- 466,484 ----
     price: $369.
  
!    GeneHunter is a genetic algorithm with a Dynamic Link Library of genetic
!    algorithm functions that may be called from programming languages such as
!    Visual Basicd or C. For non-programmers, GeneHunter also includes an
!    Exceld Add-in program which allows the user to solve an optimization
!    problem from an Excel spreadsheet. 
! 
!    Contact:
!    Ward Systems Group, Inc.;
!    Executive Park West;
!    5 Hillcrest Drive;
!    Frederick, MD 21702;
!    USA;
!    Phone: 301 662-7950; FAX: 301 662-5666.
!    email: WardSystems@msn.com
!    URL: http://www.wardsystems.com
     Contact us for a free demo diskette and Consumer's Guide to Neural
     Networks. 
***************
*** 522,528 ****
  +++++++++++++
  
!    Name: Neuralyst Version 1.4; Company: Cheshire Engineering Corporation;
!    Address: 650 Sierra Madre Villa, Suite 201, Pasedena CA 91107; Phone:
!    818-351-0209; Fax: 818-351-8645;
  
     Basic capabilities: training of backpropogation neural nets. Operating
--- 526,534 ----
  +++++++++++++
  
!    Name:  Neuralyst Version 1.4;
!    Company:  Cheshire Engineering Corporation;
!    Address:  650 Sierra Madre Villa, Suite 201, Pasedena CA 91107;
!    Phone:    818-351-0209;
!    Fax:      818-351-8645;
  
     Basic capabilities: training of backpropogation neural nets. Operating
***************
*** 529,540 ****
     system: Windows or Macintosh running Microsoft Excel Spreadsheet.
     Neuralyst is an add-in package for Excel. Approx. price: $195 for windows
!    or Mac. Comments: A simple model that is easy to use. Integrates nicely
!    into Microsoft Excel. Allows user to create, train, and run backprop ANN
!    models entirely within an Excel spreadsheet. Provides macro functions
!    that can be called from Excel macro's, allowing you to build a custom
!    Window's interface using Excel's macro language and Visual Basic tools.
!    The new version 1.4 includes a genetic algorithm to guide the training
!    process. A good bargain to boot. (Comments by Duane Highley, a user and
!    NOT the program developer. dhighley@ozarks.sgcl.lib.mo.us) 
  
  12. NeuFuz4
--- 535,539 ----
     system: Windows or Macintosh running Microsoft Excel Spreadsheet.
     Neuralyst is an add-in package for Excel. Approx. price: $195 for windows
!    or Mac. 
  
  12. NeuFuz4
***************
*** 541,573 ****
  +++++++++++
  
!    Name: NeuFuz4 Company: National Semiconductor Corporation Address: 2900
!    Semiconductor Drive, Santa Clara, CA, 95052, or: Industriestrasse 10,
!    D-8080 Fuerstenfeldbruck, Germany, or: Sumitomo Chemical Engineering
!    Center, Bldg. 7F 1-7-1, Nakase, Mihama-Ku, Chiba-City, Ciba Prefecture
!    261, JAPAN, or: 15th Floor, Straight Block, Ocean Centre, 5 Canton Road,
!    Tsim Sha Tsui East, Kowloon, Hong Kong, Phone: (800) 272-9959 (Americas),
!    : 011-49-8141-103-0 Germany : 0l1-81-3-3299-7001 Japan : (852) 737-1600
!    Hong Kong Email: neufuz@esd.nsc.com (Neural net inquiries only) URL:
!    http://www.commerce.net/directories/participants/ns/home.html Basic
!    capabilities: Uses backpropagation techniques to initially select fuzzy
!    rules and membership functions. The result is a fuzzy associative memory
!    (FAM) which implements an approximation of the training data. Operating
!    Systems: 486DX-25 or higher with math co-processor DOS 5.0 or higher with
!    Windows 3.1, mouse, VGA or better, minimum 4 MB RAM, and parallel port.
!    Approx. price : depends on version - see below. Comments : Not for the
!    serious Neural Network researcher, but good for a person who has little
!    understanding of Neural Nets - and wants to keep it that way. The systems
!    are aimed at low end controls applications in automotive, industrial, and
!    appliance areas. NeuFuz is a neural-fuzzy technology which uses
!    backpropagation techniques to initially select fuzzy rules and membership
!    functions. Initial stages of design using NeuFuz technology are performed
!    using training data and backpropagation. The result is a fuzzy
!    associative memory (FAM) which implements an approximation of the
!    training data. By implementing a FAM, rather than a multi-layer
!    perceptron, the designer has a solution which can be understood and tuned
!    to a particular application using Fuzzy Logic design techniques. There
!    are several different versions, some with COP8 Code Generator (COP8 is
!    National's family of 8-bit microcontrollers) and COP8 in-circuit emulator
!    (debug module). 
  
  13. Cortex-Pro
--- 540,582 ----
  +++++++++++
  
!          Name: NeuFuz4
!       Company: National Semiconductor Corporation
!       Address: 2900 Semiconductor Drive, Santa Clara, CA, 95052,
!            or: Industriestrasse 10, D-8080 Fuerstenfeldbruck, Germany,
!            or: Sumitomo Chemical Engineering Center, Bldg. 7F 1-7-1, Nakase,
!                  Mihama-Ku, Chiba-City, Ciba Prefecture 261, JAPAN,
!            or: 15th Floor, Straight Block, Ocean Centre, 5 Canton Road, Tsim
!                  Sha Tsui East, Kowloon, Hong Kong,
!         Phone: (800) 272-9959    (Americas),
!              : 011-49-8141-103-0 Germany
!              : 0l1-81-3-3299-7001 Japan
!              : (852) 737-1600 Hong Kong
!         Email: neufuz@esd.nsc.com (Neural net inquiries only)
!           URL: http://www.commerce.net/directories/participants/ns/home.html
! 
!    Basic capabilities:
!      Uses backpropagation techniques to initially select fuzzy rules
!      and membership functions. The result is a fuzzy associative memory (FAM)
!      which implements an approximation of the training data.
!    Operating Systems: 486DX-25 or higher with math co-processor
!                       DOS 5.0 or higher  with Windows 3.1, mouse,
!                       VGA or better, minimum 4 MB RAM, and parallel port.
!    Approx. price    : depends on version - see below.
!    Comments         :
!       Not for the serious Neural Network researcher, but good for a person
!       who has little understanding of Neural Nets - and wants to keep it that
!       way. The systems are aimed at low end controls applications in
!       automotive, industrial, and appliance areas.  NeuFuz is a neural-fuzzy
!       technology which uses backpropagation techniques to initially select
!       fuzzy rules and membership functions.  Initial stages of design using
!       NeuFuz technology are performed using training data and
!       backpropagation. The result is a fuzzy associative memory (FAM) which
!       implements an approximation of the training data.  By implementing a
!       FAM, rather than a multi-layer perceptron, the designer has a solution
!       which can be understood and tuned to a particular application using
!       Fuzzy Logic design techniques.
!       There are several different versions, some with COP8 Code Generator
!       (COP8 is National's family of 8-bit microcontrollers) and
!       COP8 in-circuit emulator (debug module).
  
  13. Cortex-Pro
***************
*** 575,581 ****
  
     Cortex-Pro information is on WWW at: 
!    http://www.neuronet.ph.kcl.ac.uk/neuronet/software/cortex/www1.html. You
!    can download a working demo from there. Contact: Michael Reiss (
!    http://www.mth.kcl.ac.uk/~mreiss/mick.html) email: <m.reiss@kcl.ac.uk>. 
  
  14. PARTEK
--- 584,591 ----
  
     Cortex-Pro information is on WWW at: 
!    http://www.neuronet.ph.kcl.ac.uk/neuronet/software/cortex/www1.html.
!    You can download a working demo from there.
!    Contact: Michael Reiss ( http://www.mth.kcl.ac.uk/~mreiss/mick.html)
!    email: <m.reiss@kcl.ac.uk>. 
  
  14. PARTEK
***************
*** 591,703 ****
     1. The PARTEK/AVB - The Analytical/Visual Base. (TM) 
  
!            * Analytical Spreadsheet (TM)
!              The Analytical Spreadsheet is a powerful and easy to use data analysis,
!              transformations, and visualization tool.  Some features include:
!                 - import native format ascii/binary data
!                 - recognition and resolution of missing data
!                 - complete set of common mathematical & statistical functions
!                 - contingency table analysis / correspondence analysis
!                 - univariate histogram analysis
!                 - extensive set of smoothing and normalization transformations
!                 - easily and quickly plot color-coded 1-D curves and histograms,
!                   2-D, 3-D, and N-D mapped scatterplots, highlighting selected
!                   patterns
!                 - Command Line (Tcl) and Graphical Interface
! 
!            * Pattern Visualization System (TM)
!              The Pattern Visualization System offers the most powerful tools for
!              visual analysis of the patterns in your data.  Some features include:
!                 - automatically maps N-D data down to 3-D for visualization of
!                   *all* of your variables at once
!                 - hard copy color Postscript output
!                 - a variety of color-coding, highlighting, and labeling options
!                   allow you to generate meaningful graphics
! 
!            * Data Filters
!              Filter out selected rows and/or columns of your data for flexible and
!              efficient cross-validation, jackknifing, bootstrapping, feature set
!              evaluation, and more.
! 
!            * Random # Generators
!              Generate random numbers from any of the following parameterized
!              distributions:
!                 - uniform, normal, exponential, gamma, binomial, poisson
! 
!            * Many distance/similarity metrics
!              Choose the appropriate distance metric for your data:
!                 - euclidean, mahalanobis, minkowski, maximum value, absolute value,
!                   shape coefficient, cosine coefficient, pearson correlation,
!                   rank correlation, kendall's tau, canberra, and bray-curtis
  

==> nn7.changes.body <==
*** nn7.oldbody.Wed Feb 28 23:00:43 1996
--- nn7.body.Thu Mar 28 23:00:34 1996
***************
*** 1,4 ****
  Archive-name: ai-faq/neural-nets/part7
! Last-modified: 1996-01-06
  URL: ftp://ftp.sas.com/pub/neural/FAQ7.html
  Maintainer: saswss@unx.sas.com (Warren S. Sarle)
--- 1,4 ----
  Archive-name: ai-faq/neural-nets/part7
! Last-modified: 1996-03-27
  URL: ftp://ftp.sas.com/pub/neural/FAQ7.html
  Maintainer: saswss@unx.sas.com (Warren S. Sarle)
***************
*** 12,54 ****
  
  Part 1: Introduction
- 
-    What is this newsgroup for? How shall it be used?
-    What is a neural network (NN)?
-    What can you do with a Neural Network and what not?
-    Who is concerned with Neural Networks?
- 
  Part 2: Learning
- 
-    What does 'backprop' mean? What is 'overfitting'?
-    Why use a bias input? Why activation functions?
-    How many hidden units should I use?
-    How many learning methods for NNs exist? Which?
-    What about Genetic Algorithms and Evolutionary Computation?
-    What about Fuzzy Logic?
-    How are NNs related to statistical methods?
- 
  Part 3: Information resources
- 
-    Good introductory literature about Neural Networks?
-    Any journals and magazines about Neural Networks?
-    The most important conferences concerned with Neural Networks?
-    Neural Network Associations?
-    Other sources of information about NNs?
- 
  Part 4: Datasets
- 
-    Databases for experimentation with NNs?
- 
  Part 5: Free software
- 
-    Freely available software packages for NN simulation?
- 
  Part 6: Commercial software
! 
!    Commercial software packages for NN simulation?
  
- Part 7: Hardware
- 
     Neural Network hardware?
  
  ------------------------------------------------------------------------
--- 12,24 ----
  
  Part 1: Introduction
  Part 2: Learning
  Part 3: Information resources
  Part 4: Datasets
  Part 5: Free software
  Part 6: Commercial software
! Part 7: Hardware, etc.
  
     Neural Network hardware?
+    Unanswered FAQs
  
  ------------------------------------------------------------------------
***************
*** 370,373 ****
--- 340,366 ----
  ------------------------------------------------------------------------
  
+ Subject: Unanswered FAQs
+ ========================
+ 
+ If you have good answers for any of these questions, please send them to the
+ FAQ maintainer at saswss@unx.sas.com. 
+ 
+  o How do NNs generalize? 
+  o What is the curse of dimensionality? 
+  o How many training cases do I need? 
+  o How many layers should be used? 
+  o How should I split the data into training and validation sets? 
+  o What are some good constructive training algorithms? 
+  o How can on-line/incremental training be done effectively? 
+  o How can I invert a network? 
+  o How can I select important input variables? 
+  o How to handle missing data? 
+  o Comparison of MLPs and RBF networks? 
+  o Should NNs be used in safety-critical applications? 
+  o My net won't learn! What should I do??? 
+  o My net won't generalize! What should I do??? 
+ 
+ ------------------------------------------------------------------------
+ 
  That's all folks (End of the Neural Network FAQ).
  
***************
*** 415,418 ****
--- 408,412 ----
   o Daniel Tauritz <dtauritz@wi.leidenuniv.nl> 
   o Wlodzislaw Duch <duch@phys.uni.torun.pl> 
+  o E. Robert Tisdale <edwin@flamingo.cs.ucla.edu> 
   o Athanasios Episcopos <EPISCOPO@icarus.som.clarkson.edu> 
   o Frank Schnorrenberg <fs0997@easttexas.tamu.edu> 
-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
