5 The Feedforward Neural Network

This chapter describes feedforward neural networks (FF networks), also known as backpropagation networks and multilayer perceptrons. Definitions, commands, and options are discussed in Section 5.1, Feedforward Network Functions and Options, and examples may be found in Section 5.2, Examples. A short tutorial introducing FF networks can be found in Section 2.5.1, Feedforward Neural Networks. Chapter 13, Changing the Neural Network Structure, describes how you can use the options and other ways to define more advanced network structures.

FF networks have a lot in common with Chapter 6, Radial Basis Function Networks. They are used for the same types of problems, and they use the same training algorithms (see Section 2.5.3, Training Feedforward and Radial Basis Function Networks).

The Neural Networks package supports the use of FF networks in three special types of problems, as follows:

Function approximation

Classification

Modeling of dynamic systems and time series

This section illustrates the first two applications. Dynamic neural network models are described in Chapter 8, Dynamic Neural Networks. However, since the dynamic neural network models are based on FF networks, it might still be interesting to examine them here.

The Neural Networks package offers several important features for FF networks, most of which are uncommon in other neural network software products. These features are listed here with links to places where more detailed descriptions are given.

Initialization: There are special initialization algorithms that give well-initialized neural networks. You can obtain an initialization with better performance from these than from one derived from a linear model. After initialization the performance is improved by the training.

Fixed parameters: You do not have to train all parameters. By keeping some of them fixed to values of your choice, you can obtain special model structures that are, for example, linear in some parameters. This is described in Section 13.2, Fixed Parameters.

Different neuron activation function: You can specify any nonlinear activation function for the neuron. This is described in Section 13.3, Select Your Own Neuron Function.

Regularization and stopped search: These techniques help you to obtain models that generalize better on new data. This is covered in Section 7.5, Regularization and Stopped Search.

Linear models: You can obtain linear models by specifying an FF network without hidden layers. The subsection Section 2.5.1, Feedforward Neural Networks discusses why this might be a good choice.

Linear model in parallel to the network: You can choose to have a linear model in parallel to the neural network by setting the option LinearPart to True in InitializeFeedForwardNet of the FF network.

Several of these features make use of a combination of numeric and symbolic capabilities of Mathematica.