6.2.1 Function Approximation in One Dimension

The following example illustrates a one-dimensional approximation problem. You can change the example by modifying the function that generates the data and then reevaluating the entire example. The result depends not only on the data and the number of chosen neuron basis functions, but also on the initialization of the parameters of the RBF network. Since the initialization is partly random, you should expect that the answers will differ each time the network is trained.

Read in the Neural Networks package and a standard add-on package.

In[1]:=

In[2]:=

It is always a good idea to look at the data. In one-dimensional problems this is especially easy. Here you can also modify the number of data samples and the function that generates the data.

Generate and look at data.

In[3]:=

The training data consist of the input data x and the output y.

Initialize an RBF network with four neurons.

In[7]:=

Out[7]=

Find some information about the network.

In[8]:=

Out[8]=

You can check the performance of the RBF network before the training. Often the initialization gives a fairly good result.

Look at the initialized RBF network on the data domain.

In[9]:=

Fit the network to the data applying three iterations.

In[10]:=

Often a warning is given at the end of the training that indicates that the training was not complete. This is quite common in connection with neural network models, especially when they involve a large number of parameters. Usually, the best thing to do is to look at the decrease in the performance index to decide whether more training iterations are necessary. In this example where only three iterations were applied, you normally would apply more so that you see that the criterion flattened out.

The trained RBF network can now be used to process new input data. The output is the approximation of the unknown true function at the point corresponding to the input.

Evaluate the RBF model at a new input value.

In[11]:=

Out[11]=

Evaluate the RBF model at several new input values.

In[12]:=

Out[12]=

By applying the network to a list of symbols you obtain a Mathematica expression describing the network.

In[13]:=

Out[13]=

You can then go on and manipulate this expression as any other Mathematica expression.

You can plot the function of the RBF network on any interval of your choice.

Plot the RBF network on the interval {-2,4}.

In[14]:=

Usually, a function estimate cannot be used outside the domain of the training data. In this case, this means that the RBF network can only be used on the interval {0,10}. Go back and check that this was the interval of the training data.

You can also use the command NetPlot to plot the function approximation together with the data. The range of the plot is set to the range of the supplied input data.

Plot the estimated function.

In[15]:=

NetPlot can be used in many ways to illustrate the quality of the trained neural network. For example, by giving the option DataFormatNetOutput you obtain a plot of the model output as a function of the given output. If the network fits the data exactly, then this plot shows a straight line with unit slope through the origin. In real applications you always have noise on your measurement, and you can only expect, if the model is good, an approximate straight line.

Plot the model output versus the data output.

In[16]:=

It might be interesting to look at the position of the basis functions. If several of them are placed on the same position, or close to another, this indicates that the number of neurons was higher than necessary, or that the initial parameters were unfavorable. In that case you can go back and initialize a new RBF network with fewer neurons. By giving the option DataFormatHiddenNeurons, you display the output values of the individual neurons versus the data.

Look at the basis functions.

In[17]:=

If you see fewer than four basis functions in the plot, then two or more of the basis functions are identical and placed on top of each other. These functions can be identified using the legend. Remember that the result will be slightly different each time you evaluate the example. If some basis functions are identical, then one of them may actually be removed without compromising the approximation quality. Assume that you want to delete the second basis function.

Remove the second neuron, plot the basis function, and plot the approximation of the new network.

In[18]:=

Out[18]=

In[19]:=

In[20]:=

By giving the option DataFormatErrorDistribution, you obtain a histogram showing the distribution of the error of the fit. This plot may indicate if you have problems with outliers.

In[21]:=

The training record can be used to evaluate the training. Section 7.8, The Training Record, shows you how to extract various kinds of information from the training record. You can also use NetPlot to plot some information.

Look at how the parameter values change during the training.

In[22]:=

Often, depending on the realization, you can see two parameters becoming very large, but with opposite signs. If this happens, then these two parameters are probably in belonging to two identical neuron basis functions. Since the basis functions are nearly identical, the effects of the huge parameters cancel each other.

Check how the parameters are stored in the RBF network.

In[23]:=

Out[23]=

Extract matrix w2 to check which basis function has large parameter values.

In[24]:=

Out[24]//MatrixForm=

By choosing DataFormatFunctionPlot, you can see how the approximation improves during the training.

Look at the function approximation after every training iteration.

In[25]:=

If you prefer, the progress can be animated as described in Section 5.2.1, Function Approximation in One Dimension, instead of being given in a graphics array.

You can also look at how the basis functions move during the training by giving the option DataFormatHiddenNeurons.

Look at the basis functions during training.

In[26]:=

From the series of neuron plots, you may see if two basis functions become equal.