Neural Network Frequently Asked Questions (FAQ)

A1) What is the comp.ai.neural-nets newsgroup for ?

There should be the following types of articles in this newsgroup:

1. Requests.
   Requests are articles of the form

   "I am looking for X"

   where X is something public like a book, an article, a piece of software. The most important about such a request is to be as specific as possible!

   If multiple different answers can be expected, the person making the request should prepare to make a summary of the answers he/she got and announce to do so with a phrase like

   "Please reply by email, I'll summarize to the group"

   at the end of the posting.

   The Subject line of the posting should then be something like

   "Request: X"

2. Questions.
   As opposed to requests, question ask for a larger piece of information or a more or less detailed explanation of something. To avoid lots of redundant traffic it is important that the poster provides with the question all information s/he already has about the subject asked and state the actual question as precise and narrow as possible.
   The poster should prepare to make a summary of the answers he/she got and announce to do so with a phrase like

   "Please reply by email, I'll summarize to the group"

   at the end of the posting.

   The Subject line of the posting should be something like

   "Question: this-and-that"

   or have the form of a question (i.e., end with a question mark)

3. Answers
   These are reactions to questions or requests. As a rule of thumb articles of type "answer" should be rare. Ideally, in most cases either the answer is too specific to be of general interest (and should thus be e-mailed to the poster) or a summary was announced with the question or request (and answers should thus be e-mailed to the poster).

   The subject lines of answers are automatically adjusted by the news software. Note that sometimes longer threads of discussion evolve from an answer to a question or request. In this case posters should change the subject line suitably as soon as the topic goes too far away from the one announced in the original subject line. You can still carry along the old subject in parentheses in the form

   "Subject: <...new subject...> (was: <...old subject...>)

4. Summaries
   In all cases of requests or questions the answers for which can be assumed to be of some general interest, the poster of the request or question shall summarize the ansers he/she received. Such a summary should be announced in the original posting of the question or request with a phrase like

   "Please answer by email, I'll summarize"

   In such a case, people who answer to a question should NOT post their answer to the newsgroup but instead mail them to the poster of the question who collects and reviews them. After about 5 to 20 days after the original posting, its poster should make the summary of answers and post it to the newsgroup.

   Some care should be invested into a summary:

   1. simple concatenation of all the answers is not enough: instead, redundancies, irrelevancies, verbosities, and errors should be filtered out (as good as possible)
   2. the answers should be separated clearly
   3. the contributors of the individual answers should be identifiable (unless they requested to remain anonymous [yes, that happens])
   4. the summary should start with the "quintessence" of the answers, as seen by the original poster
   5. A summary should, when posted, clearly be indicated to be one by giving it a Subject line starting with "SUMMARY:"
   Note that a good summary is pure gold for the rest of the newsgroup community, so summary work will be most appreciated by all of us. (Good summaries are more valuable than any moderator ! :-> )

5. Announcements
   Some articles never need any public reaction. These are called announcements (for instance for a workshop, conference or the availability of some technical report or software system).

   Announcements should be clearly indicated to be such by giving them a subject line of the form

   "Announcement: this-and-that"

6. Reports
   Sometimes people spontaneously want to report something to the newsgroup. This might be special experiences with some software, results of own experiments or conceptual work, or especially interesting information from somewhere else.

   Reports should be clearly indicated to be such by giving them a subject line of the form

   "Report: this-and-that"

7. Discussions
   An especially valuable possibility of Usenet is of course that of discussing a certain topic with hundreds of potential participants. All traffic in the newsgroup that can not be subsumed under one of the above categories should belong to a discussion.

   If somebody explicitly wants to start a discussion, he/she can do so by giving the posting a subject line of the form

   "Subject: Discussion: this-and-that"

   It is quite difficult to keep a discussion from drifting into chaos, but, unfortunately, as many many other newsgroups show there seems to be no secure way to avoid this. On the other hand, comp.ai.neural-nets has not had many problems with this effect in the past, so let's just go and hope... :->

A2) What is a neural network (NN) ?

A vague description is as follows:

An ANN is a network of many very simple processors ("units"), each possibly having a (small amount of) local memory. The units are connected by unidirectional communication channels ("connections"), which carry numeric (as opposed to symbolic) data. The units operate only on their local data and on the inputs they receive via the connections.

The design motivation is what distinguishes neural networks from other mathematical techniques:

A neural network is a processing device, either an algorithm, or actual hardware, whose design was motivated by the design and functioning of human brains and components thereof.

Neural networks normally have great potential for parallelism, since the computations of the components are independent of each other.

A3) What can you do with a Neural Network and what not ?

In practice, NNs are especially useful for mapping problems which are tolerant of a high error rate, have lots of example data available, but to which hard and fast rules can not easily be applied. NNs are, at least today, difficult to apply successfully to problems that concern manipulation of symbols and memory.

A4) Who is concerned with Neural Networks ?

• Computer scientists want to find out about the properties of non-symbolic information processing with neural nets and about learning systems in general.

• Engineers of many kinds want to exploit the capabilities of neural networks on many areas (e.g. signal processing) to solve their application problems.

• Cognitive scientists view neural networks as a possible apparatus to describe models of thinking and conscience (High-level brain function).

• Neuro-physiologists use neural networks to describe and explore medium-level brain function (e.g. memory, sensory system, motorics).

• Physicists use neural networks to model phenomena in statistical mechanics and for a lot of other tasks.

• Biologists use Neural Networks to interpret nucleotide sequences.

• Philosophers and some other people may also be interested in Neural Networks for various reasons.

A5) What does 'backprop' mean ?

"You are almost not knowing what you are actually doing when using backpropagation" :-)

It is for the training of layered (i.e., nodes are grouped in layers) feedforward (i.e., the arcs joining nodes are unidirectional, and there are no cycles) nets.

Back-propagation needs a teacher that knows the correct output for any input ("supervised learning") and uses gradient descent on the error (as provided by the teacher) to train the weights. The activation function is (usually) a sigmoidal (i.e., bounded above and below, but differentiable) function of a weighted sum of the nodes inputs.

The use of a gradient descent algorithm to train its weights makes it slow to train; but being a feedforward algorithm, it is quite rapid during the recall phase.

Literature:

Rumelhart, D. E. and McClelland, J. L. (1986): Parallel Distributed Processing: Explorations in the Microstructure of Cognition (volume 1, pp 318-362). The MIT Press.

A6) How many learning methods for NNs exist ? Which ?

The main categorization of these methods is the distiction of supervised from unsupervised learning:

• In supervised learning, there is a "teacher" who in the learning phase "tells" the net how well it performs ("reinforcement learning") or what the correct behavior would have been ("fully supervised learning").
• In unsupervised learning the net is autonomous: it just looks at the data it is presented with, finds out about some of the properties of the data set and learns to reflect these properties in its output. What exactly these properties are, that the network can learn to recognise, depends on the particular network model and learning method.

Now here is the list, just giving some names:

1. UNSUPERVISED LEARNING (i.e. without a "teacher"):
   • Feedback Nets:
     • Additive Grossberg (AG)
     • Shunting Grossberg (SG)
     • Binary Adaptive Resonance Theory (ART1)
     • Analog Adaptive Resonance Theory (ART2, ART2a)
     • Discrete Hopfield (DH)
     • Continuous Hopfield (CH)
     • Discrete Bidirectional Associative Memory (BAM)
     • Temporal Associative Memory (TAM)
     • Adaptive Bidirectional Associative Memory (ABAM)
     • Kohonen Self-organizing Map (SOM)
     • Kohonen Topology-preserving Map (TPM)
   • Feedforward-only Nets:
     • Learning Matrix (LM)
     • Driver-Reinforcement Learning (DR)
     • Linear Associative Memory (LAM)
     • Optimal Linear Associative Memory (OLAM)
     • Sparse Distributed Associative Memory (SDM)
     • Fuzzy Associative Memory (FAM)
     • Counterprogation (CPN)
2. SUPERVISED LEARNING (i.e. with a "teacher"):
   • Feedback Nets:
     • Brain-State-in-a-Box (BSB)
     • Fuzzy Congitive Map (FCM)
     • Boltzmann Machine (BM)
     • Mean Field Annealing (MFT)
     • Recurrent Cascade Correlation (RCC)
     • Learning Vector Quantization (LVQ)
   • Feedforward-only Nets:
     • Perceptron
     • Adaline, Madaline
     • Backpropagation (BP)
     • Cauchy Machine (CM)
     • Adaptive Heuristic Critic (AHC)
     • Time Delay Neural Network (TDNN)
     • Associative Reward Penalty (ARP)
     • Avalanche Matched Filter (AMF)
     • Backpercolation (Perc)
     • Artmap
     • Adaptive Logic Network (ALN)
     • Cascade Correlation (CasCor)

A7) What about Genetic Algorithms ?

A GA is an optimization program
that starts with some encoded procedure, (Creation of Life)
mutates it stochastically, (Get cancer or so)
and uses a selection process (Darwinism)
to prefer the mutants with high fitness
and perhaps a recombination process (Make babies)
to combine properties of (preferably) the succesful mutants.

A8) What about Fuzzy Logic ?

A9) Good introductory literature about Neural Networks ?

1. The best:
   • Hecht-Nielsen, R. (1990). Neurocomputing. Addison Wesley.
     Comments: "A good book", "comprises a nice historical overview and a chapter about NN hardware. Well structured prose. Makes important concepts clear."

   • Hertz, J., Krogh, A., and Palmer, R. (1991). Introduction to the Theory of Neural Computation. Addison-Wesley: Redwood City, California. ISBN 0-201-50395-6 (hardbound) and 0-201-51560-1 (paperbound)
     Comments: "My first impression is that this one is by far the best book on the topic. And it's below $30 for the paperback."; "Well written, theoretical (but not overwhelming)"; It provides a good balance of model development, computational algorithms, and applications. The mathematical derivations are especially well done"; "Nice mathematical analysis on the mechanism of different learning algorithms"; "It is NOT for mathematical beginner. If you don't have a good grasp of higher level math, this book can be really tough to get through."

2. Books for the beginner:
   • Aleksander, I. and Morton, H. (1990). An Introduction to Neural Computing. Chapman and Hall. (ISBN 0-412-37780-2).
     Comments: "This book seems to be intended for the first year of university education."

   • Beale, R. and Jackson, T. (1990). Neural Computing, an Introduction. Adam Hilger, IOP Publishing Ltd : Bristol. (ISBN 0-85274-262-2).
     Comments: "It's clearly written. Lots of hints as to how to get the adaptive models covered to work (not always well explained in the original sources). Consistent mathematical terminology. Covers perceptrons, error-backpropagation, Kohonen self-org model, Hopfield type models, ART, and associative memories."

   • Dayhoff, J. E. (1990). Neural Network Architectures: An Introduction. Van Nostrand Reinhold: New York.
     Comments: "Like Wasserman's book, Dayhoff's book is also very easy to understand".

   • McClelland, J. L. and Rumelhart, D. E. (1988). Explorations in Parallel Distributed Processing: Computational Models of Cognition and Perception (software manual). The MIT Press.
     Comments: "Written in a tutorial style, and includes 2 diskettes of NN simulation programs that can be compiled on MS-DOS or Unix (and they do too !)"; "The programs are pretty reasonable as an introduction to some of the things that NNs can do."; "There are two editions of this book. One comes with disks for the IBM PC, the other comes with disks for the Macintosh".

   • McCord Nelson, M. and Illingworth, W.T. (1990). A Practical Guide to Neural Nets. Addison-Wesley Publishing Company, Inc. (ISBN 0-201-52376-0).
     Comments: "No formulas at all( ==> no good)"; "It does not have much detailed model development (very few equations), but it does present many areas of application. It includes a chapter on current areas of research. A variety of commercial applications is discussed in chapter 1. It also includes a program diskette with a fancy graphical interface (unlike the PDP diskette)".

   • Orchard, G.A. & Phillips, W.A. (1991). Neural Computation: A Beginner's Guide. Lawrence Earlbaum Associates: London.
     Comments: "Short user-friendly introduction to the area, with a non-technical flavour. Apparently accompanies a software package, but I haven't seen that yet".

   • Wasserman, P. D. (1989). Neural Computing: Theory & Practice. Van Nostrand Reinhold: New York. (ISBN 0-442-20743-3).
     Comments: "Wasserman flatly enumerates some common architectures from an engineer's perspective ('how it works') without ever addressing the underlying fundamentals ('why it works') - important basic concepts such as clustering, principal components or gradient descent are not treated. It's also full of errors, and unhelpful diagrams drawn with what appears to be PCB board layout software from the '70s. For anyone who wants to do active research in the field I consider it quite inadequate"; "Okay, but too shallow"; "Quite easy to understand"; "The best bedtime reading for Neural Networks. I have given this book to numerous collegues who want to know NN basics, but who never plan to implement anything. An excellent book to give your manager."

3. The classics:
   • Kohonen, T. (1984). Self-organization and Associative Memory. Springer-Verlag: New York. (2nd Edition: 1988; 3rd edition: 1989).
     Comments: "The section on Pattern mathematics is excellent."

   • Rumelhart, D. E. and McClelland, J. L. (1986). Parallel Distributed Processing: Explorations in the Microstructure of Cognition (volumes 1 & 2). The MIT Press.
     Comments: "As a computer scientist I found the two Rumelhart and McClelland books really heavy going and definitely not the sort of thing to read if you are a beginner."; "It's quite readable, and affordable (about $65 for both volumes)."; "THE Connectionist bible.".

4. Introductory journal articles:
   • Hinton, G. E. (1989). Connectionist learning procedures. Artificial Intelligence, Vol. 40, pp. 185--234.
     Comments: "One of the better neural networks overview papers, although the distinction between network topology and learning algorithm is not always very clear. Could very well be used as an introduction to neural networks."

   • Knight, K. (1990). Connectionist, Ideas and Algorithms. Communications of the ACM. November 1990. Vol.33 nr.11, pp 59-74.
     Comments:"A good article, while it is for most people easy to find a copy of this journal."

   • Kohonen, T. (1988). An Introduction to Neural Computing. Neural Networks, vol. 1, no. 1. pp. 3-16.
     Comments: "A general review".

5. Not-quite-so-introductory literature:
   • Anderson, J. A. and Rosenfeld, E. (Eds). (1988). Neurocomputing: Foundations of Research. The MIT Press: Cambridge, MA.
     Comments: "An expensive book, but excellent for reference. It is a collection of reprints of most of the major papers in the field."

   • Anderson, J. A., Pellionisz, A. and Rosenfeld, E. (Eds). (1990). Neurocomputing 2: Directions for Research. The MIT Press: Cambridge, MA.
     Comments: "The sequel to their well-known Neurocomputing book."

   • Caudill, M. and Butler, C. (1990). Naturally Intelligent Systems. MIT Press: Cambridge, Massachusetts. (ISBN 0-262-03156-6).
     Comments: "I guess one of the best books I read"; "May not be suited for people who want to do some research in the area".

   • Khanna, T. (1990). Foundations of Neural Networks. Addison-Wesley: New York.
     Comments: "Not so bad (with a page of erroneous formulas (if I remember well), and #hidden layers isn't well described)."; "Khanna's intention in writing his book with math analysis should be commended but he made several mistakes in the math part".

   • Levine, D. S. (1990). Introduction to Neural and Cognitive Modeling. Lawrence Erlbaum: Hillsdale, N.J.
     Comments: "Highly recommended".

   • Lippmann, R. P. (April 1987). An introduction to computing with neural nets. IEEE Acoustics, Speech, and Signal Processing Magazine. vol. 2, no. 4, pp 4-22.
     Comments: "Much acclaimed as an overview of neural networks, but rather inaccurate on several points. The categorization into binary and continuous- valued input neural networks is rather arbitrary, and may work confusing for the unexperienced reader. Not all networks discussed are of equal importance."

   • Maren, A., Harston, C. and Pap, R., (1990). Handbook of Neural Computing Applications. Academic Press. ISBN: 0-12-471260-6. (451 pages).
     Comments: "They cover a broad area"; "Introductory with suggested applications implementation".

   • Pao, Y. H. (1989). Adaptive Pattern Recognition and Neural Networks Addison-Wesley Publishing Company, Inc. (ISBN 0-201-12584-6).
     Comments: "An excellent book that ties together classical approaches to pattern recognition with Neural Nets. Most other NN books do not even mention conventional approaches."

   • Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, vol 323 (9 October), pp. 533-536.
     Comments: "Gives a very good potted explanation of backprop NN's. It gives sufficient detail to write your own NN simulation."

   • Simpson, P. K. (1990). Artificial Neural Systems: Foundations, Paradigms, Applications and Implementations. Pergamon Press: New York.
     Comments: "Contains a very useful 37 page bibliography. A large number of paradigms are presented. On the negative side the book is very shallow. Best used as a complement to other books".

   • Zeidenberg. M. (1990). Neural Networks in Artificial Intelligence. Ellis Horwood, Ltd., Chichester.
     Comments: "Gives the AI point of view".

   • Zornetzer, S. F., Davis, J. L. and Lau, C. (1990). An Introduction to Neural and Electronic Networks. Academic Press. (ISBN 0-12-781881-2).
     Comments: "Covers quite a broad range of topics (collection of articles/papers )."; "Provides a primer-like introduction and overview for a broad audience, and employs a strong interdisciplinary emphasis".

A10) Any journals and magazines about Neural Networks ?

A11) The most important conferences concerned with Neural Networks ?

• Dedicated Neural Network Conferences:
  1. Neural Information Processing Systems (NIPS) Annually in Denver, Colorado; late November or early December
  2. International Joint Conference on Neural Networks (IJCNN) co-sponsored by INNS and IEEE
  3. Annual Conference on Neural Networks (ACNN)
  4. International Conference on Artificial Neural Networks (ICANN) Annually in Europe(?), 1992 in Brighton
  5. Major conference of European Neur. Netw. Soc. (ENNS)
• Other Conferences
  1. International Joint Conference on Artificial Intelligence (IJCAI)
  2. Intern. Conf. on Acustics, Speech and Signal Processing (ICASSP)
  3. Annual Conference of the Cognitive Science Society
  4. [Vision Conferences?]
• Pointers to Conferences
  1. The journal "Neural Networks" has a long list of conferences, workshops and meetings in each issue. This is quite interdisciplinary.
  2. There is a regular posting on comp.ai.neural-nets from Paultje Bakker: "Upcoming Neural Network Conferences", which lists names, dates, locations, contacts, and deadlines.

A12) Neural Network Associations ?

1. International Neural Network Society (INNS).
   INNS membership includes subscription to "Neural Networks", the official journal of the society.
   Membership is $55 for non-students and $45 for students per year.
   Address: INNS Membership, P.O. Box 491166, Ft. Washington, MD 20749.
   
2. International Student Society for Neural Networks (ISSNNets).
   Membership is $5 per year.
   Address: ISSNNet, Inc., P.O. Box 15661, Boston, MA 02215 USA.
   
3. Women In Neural Network Research and technology (WINNERS).
   Address: WINNERS, c/o Judith Dayhoff, 11141 Georgia Ave., Suite 206, Wheaton, MD 20902. Telephone: 301-933-9000.
   
4. European Neural Network Society (ENNS)
5. Japanese Neural Network Society (JNNS)
   Address: Japanese Neural Network Society Department of Engineering, Tamagawa University, 6-1-1, Tamagawa Gakuen, Machida City, Tokyo, 194 JAPAN Phone: +81 427 28 3457, Fax: +81 427 28 3597
6. Association des Connexionnistes en THese (ACTH)
   (the French Student Association for Neural Networks)
   Membership is 100 FF per year
   Activities : newsletter, conference (every year), list of members...
   Address : ACTH - Le Castelnau R2 23 avenue de la Galline 34170 Castelnau-le-Lez FRANCE
   Contact : jdmuller@vnet.ibm.com
7. Neurosciences et Sciences de l'Ingenieur (NSI)
   Biology & Computer Science
   Activity : conference (every year)
   Address : NSI - TIRF / INPG 46 avenue Felix Viallet 38031 Grenoble Cedex FRANCE

A13) Other sources of information about NNs ?

A14) Freely available software packages for NN simulation ?

A15) Commercial software packages for NN simulation ?

1. nn/xnn
   Name: nn/xnn
   Company: Neureka ANS
   Address: Klaus Hansens vei 31B, 5037 Solheimsviken, NORWAY
   Phone:+47-55544163 / +47-55201548
   Email:arnemo@eik.ii.uib.no
   Basic capabilities:

   Neural network development tool. nn is a language for specification of neural network simulators. Produces C-code and executables for the specified models, therefore ideal for application development. xnn is a graphical front-end to nn and the simulation code produced by nn. Gives graphical representations in a number of formats of any variables during simulation run-time. Comes with a number of pre-implemented models, including: Backprop (several variants), Self Organizing Maps, LVQ1, LVQ2, Radial Basis Function Networks, Generalized Regression Neural Networks, Jordan nets, Elman nets, Hopfield, etc.

   Operating system: nn: UNIX or MS-DOS, xnn: UNIX/X-windows
   System requirements: 10 Mb HD, 2 Mb RAM
   Approx. price: USD 2000,-

2. BrainMaker
   Name: BrainMaker, BrainMaker Pro
   Company: California Scientific Software
   Address: 10024 Newtown rd, Nevada City, CA, 95959 USA
   Phone,Fax: 916 478 9040, 916 478 9041
   Email: calsci!mittmann@gvgpsa.gvg.tek.com (flakey connection)
   Basic capabilities: train backprop neural nets
   Operating system: DOS, Windows, Mac
   System requirements: Uses XMS or EMS for large models(PCs only): Pro version
   Approx. price: $195, $795
   • BrainMaker Pro 3.0 (DOS/Windows) $795
     Gennetic Training add-on $250
   • BrainMaker 3.0 (DOS/Windows/Mac) $195
     Network Toolkit add-on $150
   • BrainMaker 2.5 Student version (quantity sales only, about $38 each)
   • BrainMaker Pro C30 Accelerator Board
     w/ 5Mb memory $9750
     w/32Mb memory $13,000
   • Intel iNNTS NN Development System $11,800
     Intel EMB Multi-Chip Board $9750
     Intel 80170 chip set $940
   • Introduction To Neural Networks book $30
   All Software has a 30 day money back guarantee, and unlimited free technical support.

   BrainMaker package includes:

   • The book Introduction to Neural Networks
   • BrainMaker Users Guide and reference manual
     300 pages , fully indexed, with tutorials, and sample Neural Networks
   • Netmaker
     Netmaker makes building and training Neural Networks easy, by importing and automatically creating BrainMaker's Neural Network files. Netmaker imports Lotus, Excel, dBase, and ASCII files.
   • BrainMaker
     Full menu and dialog box interface, runs Backprop at 750,000 cps on a 33Mhz 486.

     Feature                 BrainMaker  Professional    Benefit
   
     User Interface
     Pull-down Menus, Dialog Boxes   {    {    easy to learn and use; all parameters
                                               saved in a file you can edit.
     Programmable Output Files       {    {    exports data in your format to
                                               spreadsheets, graphics packages, etc.
     Editing in BrainMaker           {    {    quickly edit data, display, network
                                               connections, and more.
     Network Progress Display             {    monitors training with a simple
                                               graphic display.
     Fact Annotation                 {    {    attaches your comments to examples
                                               for display and printing.
     Printer Support                 {    {    HP LaserJet, DeskJet, InkJet,
                                               IBM Proprinter, Epson, etc.
     NetPlotter                      T    {    see how the input correlates with
                                               your output.
     Graphics Built In                    {    shows trends, cycles, network
                                               responses, statistics, etc.;
                                               see it on screen, plotter, or printer.
     Dynamic Data Exchange                {    puts your network in other windows
                                               programs
   
     Performance
     Binary Mode                     T    {    uses binary files for greater speed.
     Batch Mode                           {    add networks to your existing
                                               programs, train while you're away.
     EMS and XMS Memory                   {    up to 8192 independent variables.
     Save Network Periodically       {    {    saves results to a file in case of
                                               power failure.
     Fastest Algorithms              {    {    750,000 connections-per-second
                                               (486/50).
     Neurons per Layer             512  32,000 more inputs: model complex data
                                               with ease.
     Number of Layers                8    8    extra hidden layers can help tackle
                                               bigger problems.
   
     Training
     Specify Parameters by Layer          {    fine-tunes performance inside the netw
     Recurrence Networks                  {    Puts feedback in your network,
                                               automates historical input.
     Prune Connections and Neurons        {    improves accuracy by trimming away
                                               excess "fat".
     Add Hidden Neurons In Training  {    {    finds best size network quickly;
                                               fully automated with Professional.
     Custom Neuron Functions         {    {    optimizes training to suit any need.
     Testing While Training          {    {    trains for best performance on new
                                               data.
     Stop training when...                {    lets you decide when network has
                                               learned well.
     Heavy Weights                        {    helps networks train.
     Hypersonic Training             T    {    trains faster with this proprietary
                                               algorithm.
   
     Analysis, Advanced Functions
     Sensitivity Analysis                 {    shows you which inputs determined
                                               your results.
     Neuron Sensitivity                   {    shows you the total effect of one
                                               input on your results.
     Global Network Analysis              {    shows how the networks reacts to
                                               your inputs overall
     Contour Analysis                     {    shows peaks and valleys of the output
                                               when two inputs change
     Data Correlator                      {    finds important data and optimum
                                               time delays.
     Error Statistics Report         {    {    check your network error rate during
                                               training.
     Print or Edit Weight Matrices   {    {    examine, customize network internals.
     Competitor                           {    ranks horses, teams, stocks, etc.
                                               in finish order.
     Run Time System                      {    C source code - make programs with
                                               your network for resale.
     Chip Support                    {    {    Intel, American Neurologics,
                                               Micro Devices.
     Genetic Training Option              G    trains variations of your design
                                               and shows you which was the best.
   
     Network Data Management Functions
     NetMaker                        {    {    spreadsheet-like data manipulation
                                               and network file creation.
     NetChecker                      {    {    checks your files for errors and
                                               inconsistencies.
     Shuffle                         {    {    mixes up the order of examples for
                                               better training.
     Binary                          T    {    converts files to binary for quicker
                                               training.
     MinMax                          {    {    finds min / max / standard deviation
                                               of data for fine-tuned results.
     Data Importation                {    {    reads data from Lotus, dBASE,
                                               Excel, ASCII, binary.
     Finacial Data                        {    reads MetaStock, and Computrack
     Data Manipulation               {    {    finds indicators, oscillators,
                                               running averages, etc.
     Cyclic Analysis                      {    checks data for periodic or cyclic
                                               behavior.
     Data Types                      {    {    uses symbolic, text, picture,
                                               and numeric data.
   
     Documentation & User Support
     User's Guide                    {    {    an application development guide
                                               and quick-start booklet.
     Introduction to Neural Networks {    {    324 pp, gets you up to date in this
                                               exciting field.
   

3. SAS Software/ Neural Net add-on
   Name: SAS Software
   Company: SAS Institute, Inc.
   Address: SAS Campus Drive, Cary, NC 27513, USA
   Phone,Fax: (919) 677-8000
   Email: saswss@unx.sas.com (Neural net inquiries only)
   Basic capabilities:

   Feedforward nets with numerous training methods and loss functions, plus statistical analogs of counterpropagation and various unsupervised architectures

   Operating system: Lots
   System requirements: Lots Uses XMS or EMS for large models(PCs only): Runs under Windows, OS/2
   Approx. price: Free neural net software, but you have to license SAS/Base software and preferably the SAS/OR, SAS/ETS, and/or SAS/STAT products.
   Comments: Oriented toward data analysis and statistical applications

4. NeuralWorks
   Name: NeuralWorks Professional II Plus (from NeuralWare)
   Company: NeuralWare Inc.
   Adress: Pittsburgh, PA 15276-9910
   Phone: (412) 787-8222
   FAX: (412) 787-8220
   

   Distributor for Europe:

   Scientific Computers GmbH
   Franzstr. 107, 52064 Aachen
   Germany
   Tel. (49) +241-26041
   Fax. (49) +241-44983
   Email. info@scientific.de
   

   Basic capabilities:

   supports over 30 different nets: backprop, art-1,kohonen, modular neural network, General regression, Fuzzy art-map, probabilistic nets, self-organizing map, lvq, boltmann, bsb, spr, etc...
   Extendable with optional package.
   ExplainNet, Flashcode (compiles net in .c code for runtime), user-defined io in c possible. ExplainNet (to eliminate extra inputs), pruning, savebest,graph.instruments like correlation, hinton diagrams, rms error graphs etc.

   Operating system : PC,Sun,IBM RS6000,Apple Macintosh,SGI,Dec,HP. System requirements: varies. PC:2MB extended memory+6MB Harddisk space. Uses windows compatible memory driver (extended). Uses extended memory.
   Approx. price : call (depends on platform)
   Comments : award winning documentation, one of the market leaders in NN software.

5. MATLAB Neural Network Toolbox (for use with Matlab 4.x)
   Contact: The MathWorks, Inc., 24 Prime Park Way,Natick, MA 01760 Phone: 508-653-1415, FAX: 508-653-2997, email: info@mathworks.com
   (Comment by Richard Andrew Miles Outerbridge, RAMO@UVPHYS.PHYS.UVIC.CA) Matlab is spreading like hotcakes (and the educational discounts are very impressive). The newest release of Matlab (4.0) answers the question "if you could only program in one language what would it be?". The neural network toolkit is worth getting for the manual alone. Matlab is available with lots of other toolkits (signal processing, optimization, etc.) but I don't use them much - the main package is more than enough. The nice thing about the Matlab approach is that you can easily interface the neural network stuff with anything else you are doing.

A16) Neural Network hardware ?

Here is a list of companies contributed by xli@computing-maths.cardiff.ac.uk:

1. HNC, INC.
   5501 Oberlin Drive
   San Diego
   California 92121
   (619) 546-8877
   and a second address at
   7799 Leesburg Pike, Suite 900
   Falls Church, Virginia
   22043
   (703) 847-6808
   Note: Australian Dist.: Unitronics
   Tel : (09) 4701443
   Contact: Martin Keye
   HNC markets:
   • 'Image Document Entry Processing Terminal' - it recognises handwritten documents and converts the info to ASCII.
   • 'ExploreNet 3000' - a NN demonstrator
   • 'Anza/DP Plus'- a Neural Net board with 25MFlop or 12.5M peak interconnects per second.

2. SAIC (Sience Application International Corporation)
   10260 Campus Point Drive
   MS 71, San Diego
   CA 92121
   (619) 546 6148
   Fax: (619) 546 6736

3. Micro Devices
   30 Skyline Drive
   Lake Mary
   FL 32746-6201
   (407) 333-4379
   MicroDevices makes MD1220 - 'Neural Bit Slice' Each of the products mentioned sofar have very different usages. Although this sounds similar to Intel's product, the architectures are not.

4. Intel Corp
   2250 Mission College Blvd
   Santa Clara, Ca 95052-8125
   Attn ETANN, Mail Stop SC9-40
   (408) 765-9235
   Intel is making an experimental chip:
   80170NW - Electrically trainable Analog Neural Network (ETANN) It has 64 'neurons' on it - almost fully internally connectted and the chip can be put in an hierarchial architecture to do 2 Billion interconnects per second.
   Support software has already been made by
   California Scientific Software
   10141 Evening Star Dr #6
   Grass Valley, CA 95945-9051
   (916) 477-7481
   Their product is called 'BrainMaker'.

5. NeuralWare, Inc
   Penn Center West
   Bldg IV Suite 227
   Pittsburgh
   PA 15276
   They only sell software/simulator but for many platforms.

6. Tubb Research Limited
   7a Lavant Street
   Peterfield
   Hampshire
   GU32 2EL
   United Kingdom
   Tel: +44 730 60256

7. Adaptive Solutions Inc
   1400 NW Compton Drive
   Suite 340
   Beaverton, OR 97006
   U. S. A.
   Tel: 503 - 690 - 1236 FAX: 503 - 690 - 1249

8. NeuroDynamX, Inc.
   4730 Walnut St., Suite 101B
   Boulder, CO 80301
   Voice: (303) 442-3539 Fax: (303) 442-2854
   Internet: techsupport@ndx.com
   NDX sells a number neural network hardware products:
   • NDX Neural Accelerators: a line of i860-based accelerator cards for the PC that give up to 45 million connections per second for use with the DynaMind neural network software.
   • iNNTS: Intel's 80170NX (ETANN) Neural Network Training System. NDX's president was one of the co-designers of this chip.

And here is an incomplete list of Neurocomputers (provided by jon@kongle.idt.unit.no (Jon Gunnar Solheim)): Overview over known Neural Computers with their newest known reference.

AAP-2

Takumi Watanabe, Yoshi Sugiyama, Toshio Kondo, and Yoshihiro Kitamura. Neural network simulation on a massively parallel cellular array processor: AAP-2. In International Joint Conference on Neural Networks, 1989.

ANNA

B.E.Boser, E.Sackinger, J.Bromley, Y.leChun, and L.D.Jackel. Hardware Requirements for Neural Network Pattern Classifiers. In IEEE Micro, 12(1), pages 32-40, February 1992.

Analog Neural Computer

Paul Mueller et al. Design and performance of a prototype analog neural computer. In Neurocomputing, 4(6):311-323, 1992.

APx -- Array Processor Accelerator

F.Pazienti. Neural networks simulation with array processors. In Advanced Computer Technology, Reliable Systems and Applications; Proceedings of the 5th Annual Computer Conference, pages 547-551. IEEE Comput. Soc. Press, May 1991. ISBN: 0-8186-2141-9.

ASP -- Associative String Processor

A.Krikelis. A novel massively associative processing architecture for the implementation artificial neural networks. In 1991 International Conference on Acoustics, Speech and Signal Processing, volume 2, pages 1057-1060. IEEE Comput. Soc. Press, May 1991.

BSP400

Jan N.H. Heemskerk, Jacob M.J. Murre, Jaap Hoekstra, Leon H.J.G. Kemna, and Patrick T.W. Hudson. The bsp400: A modular neurocomputer assembled from 400 low-cost microprocessors. In International Conference on Artificial Neural Networks. Elsevier Science, 1991.

BLAST

J.G.Elias, M.D.Fisher, and C.M.Monemi. A multiprocessor machine for large-scale neural network simulation. In IJCNN91-Seattle: International Joint Conference on Neural Networks, volume 1, pages 469-474. IEEE Comput. Soc. Press, July 1991. ISBN: 0-7883-0164-1.

CNAPS Neurocomputer

H.McCartor Back Propagation Implementation on the Adaptive Solutions CNAPS Neurocomputer. In Advances in Neural Information Processing Systems, 3, 1991.

MA16 -- Neural Signal Processor

U.Ramacher, J.Beichter, and N.Bruls. Architecture of a general-purpose neural signal processor. In IJCNN91-Seattle: International Joint Conference on Neural Networks, volume 1, pages 443-446. IEEE Comput. Soc. Press, July 1991. ISBN: 0-7083-0164-1.

Mindshape

Jan N.H. Heemskerk, Jacob M.J. Murre Arend Melissant, Mirko Pelgrom, and Patrick T.W. Hudson. Mindshape: a neurocomputer concept based on a fractal architecture. In International Conference on Artificial Neural Networks. Elsevier Science, 1992.

mod 2

Michael L. Mumford, David K. Andes, and Lynn R. Kern. The mod 2 neurocomputer system design. In IEEE Transactions on Neural Networks, 3(3):423-433, 1992.

NERV

R.Hauser, H.Horner, R. Maenner, and M.Makhaniok. Architectural Considerations for NERV - a General Purpose Neural Network Simulation System. In Workshop on Parallel Processing: Logic, Organization and Technology -- WOPPLOT 89, pages 183-195. Springer Verlag, Mars 1989. ISBN: 3-5405-5027-5.

NP -- Neural Processor

D.A.Orrey, D.J.Myers, and J.M.Vincent. A high performance digital processor for implementing large artificial neural networks. In Proceedings of of the IEEE 1991 Custom Integrated Circuits Conference, pages 16.3/1-4. IEEE Comput. Soc. Press, May 1991. ISBN: 0-7883-0015-7.

RAP -- Ring Array Processor

N.Morgan, J.Beck, P.Kohn, J.Bilmes, E.Allman, and J.Beer. The ring array processor: A multiprocessing peripheral for connectionist applications. In Journal of Parallel and Distributed Computing, pages 248-259, April 1992.

RENNS -- REconfigurable Neural Networks Server

O.Landsverk, J.Greipsland, J.A.Mathisen, J.G.Solheim, and L.Utne. RENNS - a Reconfigurable Computer System for Simulating Artificial Neural Network Algorithms. In Parallel and Distributed Computing Systems, Proceedings of the ISMM 5th International Conference, pages 251-256. The International Society for Mini and Microcomputers - ISMM, October 1992. ISBN: 1-8808-4302-1.

SMART -- Sparse Matrix Adaptive and Recursive Transforms

P.Bessiere, A.Chams, A.Guerin, J.Herault, C.Jutten, and J.C.Lawson. From Hardware to Software: Designing a ``Neurostation''. In VLSI design of Neural Networks, pages 311-335, June 1990.

SNAP -- Scalable Neurocomputer Array Processor

E.Wojciechowski. SNAP: A parallel processor for implementing real time neural networks. In Proceedings of the IEEE 1991 National Aerospace and Electronics Conference; NAECON-91, volume 2, pages 736-742. IEEE Comput.Soc.Press, May 1991.

Toroidal Neural Network Processor

S.Jones, K.Sammut, C.Nielsen, and J.Staunstrup. Toroidal Neural Network: Architecture and Processor Granularity Issues. In VLSI design of Neural Networks, pages 229-254, June 1990.

SMART and SuperNode

P. Bessiere, A. Chams, and P. Chol. MENTAL : A virtual machine approach to artificial neural networks programming. In NERVES, ESPRIT B.R.A. project no 3049, 1991. (The report archived on neuroprose}

EMMA-2

R.Battiti, L.M.Briano, R.Cecinati, A.M.Colla, and P.Guido. An application oriented development environment for Neural Net models on multiprocessor Emma-2. In Silicon Architectures for Neural Nets; Proceedings for the IFIP WG.10.5 Workshop, pages 31-43. North Holland, November 1991. ISBN: 0-4448-9113-7.

iPSC/860 Hypercube

D.Jackson, and D.Hammerstrom Distributing Back Propagation Networks Over the Intel iPSC/860 Hypercube In IJCNN91-Seattle: International Joint Conference on Neural Networks, volume 1, pages 569-574. IEEE Comput. Soc. Press, July 1991. ISBN: 0-7083-0164-1.

SCAP -- Systolic/Cellular Array Processor

Wei-Ling L., V.K.Prasanna, and K.W.Przytula. Algorithmic Mapping of Neural Network Models onto Parallel SIMD Machines. In IEEE Transactions on Computers, 40(12), pages 1390-1401, December 1991. ISSN: 0018-9340.

A18) Databases for experimentation with NNs ?

Acknowledgements: