5.1.5 LinearizeNet and NeuronDelete

The commands LinearizeNet and NeuronDelete modify the structure of an existing network.

In many situations, it is interesting to linearize a nonlinear function at some point. FF networks can be linearized using LinearizeNet.

Linearize a feedforward network.

LinearizeNet returns a linear model in the form of an FF network without any hidden neurons as described in Section 2.5.1, Feedforward Neural Networks.

The point of the linearization, x, should be a list of real numbers of length equal to the number of inputs of the neural network.

The linear network corresponds to a first-order Taylor expansion of the original network in the linearization point.

Sometimes it might be of interest to remove parts of an existing network. NeuronDelete can be used to remove outputs, inputs, hidden neurons, or a linear submodel.

You can also remove individual parameters by setting their numerical values to zero and excluding them from the training, as described in Section 13.2, Fixed Parameters.

Delete neurons in an existing network.

The argument pos indicates which part of the network should be deleted in the following way:

{0,0}: removes the linear submodel.

{0,m}: removes input m.

{n,m}: removes neuron m in hidden layer n.

{n,m}: removes output m if n == number of hidden layers + 1.

The argument pos can also be a list where each element follows these rules.

NeuronDelete can be used to obtain the values of the hidden neurons of a network; if all outputs are removed, then a network is returned with outputs equal to the last hidden layer of the initial network. The output nonlinearity is set to the neuron function used in the initial network.

If input data is submitted, then the parameters of the layer following the removed neuron are adjusted so that the new network approximates the original one as well as possible. The least-squares algorithm is used for this.

There is no adjustment of the parameters if an output is removed.

If a neuron in the last hidden layer is removed, then the parameters in the linear submodel are also included in the parameter adjustment. If the linear submodel is removed, then the parameters in the last layer are adjusted.