13.3.2 The Neuron Function in a Feedforward Network

You can set the neuron activation function to any smooth function. Some of the commonly used functions are demonstrated here.

Load the Neural Networks package.

In[1]:=

Sigmoid is the default activation function.

Plot the sigmoid.

In[2]:=

The sigmoid function is also recommended as the output of an FF network when you work on classification problems. Setting OutputNonlinearity Sigmoid during initialization of an FF network will set the output of the network to Sigmoid. Sigmoid is used because it saturates to zero or one, which are the values used to indicate membership in a class. See Section 3.2 Package Conventions.

Another common choice of activation function is the hyperbolic tangent function.

Plot the hyperbolic tangent.

In[3]:=

The hyperbolic tangent and sigmoid functions are equivalent when used in the hidden neurons because there is a similarity transformation for the parameters that takes an FF network with one of the activation functions to the other. It does, however, make a difference which function you apply to the output with the OutputNonlinearity option.

An interesting alternative neuron function is the saturated linear activation function. It is linear between -1 and 1 but saturates at these values for large positive or negative numbers.

Plot the SaturatedLinear function.

In[4]:=

This activation function gives a local linear model that might be of interest in many situations.

Another possibility is the inverse tangent function.

Plot the inverse tangent.

In[5]:=

The following demonstrates the use of the SaturatedLinear function in an FF network model.

Generate data and plot the actual function to be modeled.

In[6]:=

Initialize an FF network with the SaturatedLinear activation function.

In[12]:=

Out[12]=

Fit the FF network to the data.

In[13]:=

Plot the result.

In[14]:=

The approximation obtained is locally linear, due to the special choice of activation function.

Try repeating this example with the other common functions. Edit the notebook and repeat the evaluations.