8.1.1 Initializing and Training Dynamic Neural Networks

There are two commands to obtain dynamic model structures. NeuralARXFit is used to model dynamic systems with input signals, and NeuralARFit is used to model time series where there is no external input signal. As the names indicate, they produce neural ARX and neural AR models as described in Section 2.6, Dynamic Neural Networks. The term neural AR(X) will be used when either a neural ARX or a neural AR model can be considered.There is no restriction on the dimensionality of the input and output signals; that is, the package supports multi-input multi-output (MIMO) models. In contrast to other neural network types in the Neural Networks package, there are no specific commands for initialization. Instead, if you only want to initialize the dynamic neural network, you use the training commands with 0 training iterations.

Both commands call NeuralFit internally (see Section 7.1, NeuralFit). Algorithmic details are described in Chapter 7 ,Training Feedforward and Radial Basis Function Networks. There are quite a few input arguments, and depending on how they and the options are chosen, a large variety of different models and algorithms may be obtained.

Initialize and train neural ARX models.

NeuralARXFit returns a list of two variables. The first variable is the trained neural ARX model, with head NeuralARX, and the second is a training record.

An existing neural ARX model can be submitted for more training by setting the argument neuralARX equal to the neural ARX model or its training record.

The format of the training data u and y is described in Section 3.2, Package Conventions. The regressor is specified as indicated in Section 2.6, Dynamic Neural Networks, the number of lagged outputs to be included in the regressor is given by , and the number of lagged inputs by . The delay is given by . For SISO systems these indices are given as three (positive) integers. For MIMO systems each one of them should be a list with one component for each input or output.

The neural network type indicated by the argument type should be either FeedForwardNet or RBFNet. The number of neurons (and layers for FF networks) is specified by nh as described in Section 5.1.1, InitializeFeedForwardNet and Section 6.1.1, InitializeRBFNet. A linear model is obtained by choosing type=FeedForwardNet and nh={}.

Time-series models are obtained with NeuralARFit in similar fashion to NeuralARXFit. The only difference is that only the number of lagged outputs have to be indicated in the call.

Initialize and estimate neural AR models.

NeuralARFit returns two arguments just as NeuralARXFit does; the first one is a model of type NeuralAR, and the second argument is the training record.

An existing neural ARX model can be submitted for more training by setting the argument neuralARX equal to the neural ARX model or its training record.

In addition to the compulsory arguments of NeuralARXFit and NeuralAR, you can also specify a number of additional arguments:

The number of training iterations to be applied in the call to NeuralFit.

Validation data uv and yv, so that the root-mean-square error (RMSE) can be computed on validation data after each training iteration. The returned trained model is the model which gives the smallest RMSE on the validation data. This can be the model at an iteration before the last one. In that case the model is obtained by stopped search, which can be useful to avoid overfitting. See Section 7.5, Regularization and Stopped Search, for more details.

NeuralARXFit and NeuralARFit have no options of their own, but since they rely on initialization and training of FF or RBF networks, you can submit any option applicable to InitializeRBFNet, or InitializeFeedForwardNet and NeuralFit.

During the training, intermediate results are displayed in a separate notebook, which is created automatically. After each training iteration you can see the value of the RMSE. The step length control parameter of the minimization algorithm is also shown; see Chapter 7, Training Feedforward and Radial Basis Function Networks. If you submit validation data to NeuralFit then you also get the RMSE on this data set.

At the end of the training, a plot is displayed showing the RMSE reduction as a function of the iteration number.

Using the various options of NeuralFit, as described in Section 7.7, Options Controlling Training Results Presentation, you can change the way the training results are presented.

There are often different warning messages given at the end of the training, providing some feedback as to the success of the training. By inspecting the error reduction plot, you can usually tell whether more iterations are necessary. Specifically, if the curve exhibits a reasonable downward slope by the end of the training, you should consider continuing the training.

The first output argument, the neuralAR(X) model, consists of two components. The first component is the FF or the RBF network, which the model is based on. The second component is a replacement rule, Regressor->{ , , }, specifying the regressor by the three indices , , and .

The second output argument of NeuralAR(X)Fit is the training record. Its head is NeuralFitRecord which contains two components. The first component is a copy of the trained neuralAR(X) model and the second component is a list of rules specifying intermediate values of parameters and the RMSE during the training. In Section 7.8, The Training Record, it is shown how you can extract the information from the training record. You can also use NetPlot to plot some information.