5.1.1 InitializeFeedForwardNet

You initialize an FF network with InitializeFeedForwardNet.

Initialize a feedforward network.

The returned network is an object with head FeedForwardNet, following the general format described in Section 3.2.3, Network Format. The FeedForwardNet and the RBFNet have one more replacement rule than the other network models. Its name is NumberOfInputs and it indicates how many inputs the network takes.

The number of inputs and outputs of the network do not need to be specified explicitly. They are instead extracted from the number of columns in the input and output data.

The argument nh should be a list of integers. The length of the list indicates the number of hidden layers in the network, and the integers indicate the number of neurons in each hidden layer. A linear model is obtained by setting it to an empty list, nh={}.

InitializeFeedForwardNet takes the following options.

Options of InitializeFeedForwardNet.

These are almost the same options for InitializationRBFNet, which is used to initialize radial basis function networks. The difference is the option BiasParameters with which you can obtain an FF network without the bias parameters indicated by b in Eq. (2.7) in Section 2.5.1, Feedforward Neural Networks. Normally, you should always include these parameters.

Another difference from the InitializationRBFNet is that some of the default values of the options are different.

The parameters of the network can be initialized in three different ways, depending on the option RandomInitialization:

False, which is the default. Then the parameters are initialized so that the slopes of neurons are placed within the domain of the input data. The exact initialization is still random, but the input data is used to choose a good range from which the parameters are chosen randomly. If there is no neuron in the output layer, then the linear parameters of the network are fitted to the output data using the least-squares algorithm. For most problems this gives a good initialization. If the model is over-parameterized, that is, if it has more parameters than necessary, the least-squares step may give very large parameter values, which can give problems in the training. In such cases LinearParameters is an alternative to False.

True, by which the parameters are initialized randomly from a uniform distribution. The range of the distribution can be set using the option InitialRange.

LinearParameters, by which the positions of the neurons are initialized in the same way as if False is used. The linear parameters are randomly chosen. This can be a good alternative if the model is overparameterized and if you intend to use regularization or stopped search in the training.

The options Regularization and FixedParameters can be set at the initialization of the network or when the network is trained with NeuralFit. In Section 7.5, Regularization and Stopped Search and Section 13.2, Fixed Parameters you can learn how to use these options.

The default neuron function is Sigmoid, but you can use the option Neuron to change it to any other differentiable function. How this is done is shown in Section 13.3, Select Your Own Neuron Function.

Depending on the initial parameter values of the FF network, it will converge to different local minima in the training. Therefore, it is best to repeat the training a couple of times with the same neural network type, but with different initializations. You get a new parameter initialization by repeating the command InitializeFeedForwardNet. Also, it should be noted that if you use the default option RandomInitializationFalse, you get a partly random initialization.