1.1 Features of This Package

The following table lists the neural network types supported by the Neural Networks package along with their typical usage. In Chapter 2, Neural Networks—A Short Tutorial, brief explanations of the different neural network types are given.

Neural network types supported by the Neural Networks package.

The functions in the package are constructed so that only the minimum amount of information has to be specified by the user. For example, the number of inputs and outputs of a network are automatically extracted from the dimensionality of the data and so they do not need to be entered explicitly.

Trained networks are contained in special objects with a head that identifies the type of network. You do not have to keep track of all of the parameters and other information contained in a neural network model; everything is contained in the network object. Extracting or changing parts of the neural network information can be done by addressing the appropriate part of the object.

Intermediate information is logged during the training of a network and returned in a special training record at the end of the training. This record can be used to analyze the training performance and to access parameter values at intermediate training stages.

The structure of feedforward and radial basis function neural network types can be modified to customize the network for your specific problem. For example, the neuron activation function can be changed to some other suitable function. You can also set some of the network parameters to predefined values and exclude them from the training.

A neural network model can be customized when the unknown function is known to have a special structure. For example, in many situations the unknown function is recognized as more nonlinear in some inputs than in others. The Neural Networks package allows you to define a model that is linear with respect to some of the inputs and nonlinear with respect to other inputs. After the neural network structure has been defined, you can proceed with the network's training as you would with a network that does not have a defined structure.

The Neural Networks package contains special initialization algorithms for the network parameters, or weights, that start the training with reasonably good performance. After this initialization, an iterative training algorithm is applied to the network and the parameter set is optimized. The special initialization makes the training much faster than a completely random choice for the parameters. This also alleviates difficulties encountered in problems with multiple local minima.

For feedforward, radial basis function, and dynamic neural networks, the weights are adjusted iteratively using -based methods. The Levenberg-Marquardt algorithm is used by default, because it is considered to be the best choice for most problems. Another feature in favor of this algorithm is that it can take advantage of a situation where a network is linear in some of its parameters. Making use of the separability of the linear and nonlinear parts of the underlying minimization problem will speed up training considerably.

For large data sets and large neural network models, the training algorithms for some types of neural networks will become computation intensive. This package reduces the computation load in two ways: (1) the expressions are optimized before numerical evaluation, thus minimizing the number of operations, and (2) the computation-intensive functions use the Compile command to send compiled code to Mathematica. Because compiled code can only work with machine-precision numbers, numerical precision will be somewhat restricted. In most practical applications this limitation will be of little significance. If you would prefer noncompiled evaluation, you could set the compiled option to false, Compiled False.