Usage of activations

Activations can either be used through an Activation layer, or through the activation argument supported by all forward layers:

from keras.layers import Activation, Dense
model.add(Dense(64))
model.add(Activation('tanh'))

This is equivalent to:

model.add(Dense(64, activation='tanh'))

You can also pass an element-wise TensorFlow/Theano/CNTK function as an activation:

from keras import backend as K
model.add(Dense(64, activation=K.tanh))

Available activations

elu

keras.activations.elu(x, alpha=1.0)

Exponential linear unit.

Arguments

• x: Input tensor.
• alpha: A scalar, slope of negative section.

Returns

The exponential linear activation: x if x > 0 andalpha * (exp(x)-1) if x < 0.

References

• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

softmax

keras.activations.softmax(x, axis=-1)

Softmax activation function.

Arguments

• x: Input tensor.
• axis: Integer, axis along which the softmax normalization is applied.

Returns

Tensor, output of softmax transformation.

Raises

• ValueError: In case dim(x) == 1.

selu

keras.activations.selu(x)

Scaled Exponential Linear Unit (SELU).

SELU is equal to: scale * elu(x, alpha), where alpha and scaleare predefined constants. The values of alpha and scale arechosen so that the mean and variance of the inputs are preservedbetween two consecutive layers as long as the weights are initializedcorrectly (see lecun_normal initialization) and the number of inputsis "large enough" (see references for more information).

Arguments

• x: A tensor or variable to compute the activation function for.

Returns

The scaled exponential unit activation: scale * elu(x, alpha).

Note

• To be used together with the initialization "lecun_normal".
• To be used together with the dropout variant "AlphaDropout".

References

• Self-Normalizing Neural Networks

softplus

keras.activations.softplus(x)

Softplus activation function.

Arguments

• x: Input tensor.

Returns

The softplus activation: log(exp(x) + 1).

softsign

keras.activations.softsign(x)

Softsign activation function.

Arguments

• x: Input tensor.

Returns

The softsign activation: x / (abs(x) + 1).

relu

keras.activations.relu(x, alpha=0.0, max_value=None, threshold=0.0)

Rectified Linear Unit.

With default values, it returns element-wise max(x, 0).

Otherwise, it follows:f(x) = max_value for x >= max_value,f(x) = x for threshold <= x < max_value,f(x) = alpha * (x - threshold) otherwise.

Arguments

• x: Input tensor.
• alpha: float. Slope of the negative part. Defaults to zero.
• max_value: float. Saturation threshold.
• threshold: float. Threshold value for thresholded activation.

Returns

A tensor.

tanh

keras.activations.tanh(x)

Hyperbolic tangent activation function.

Arguments

• x: Input tensor.

Returns

The hyperbolic activation:tanh(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x))

sigmoid

keras.activations.sigmoid(x)

Sigmoid activation function.

Arguments

• x: Input tensor.

Returns

The sigmoid activation: 1 / (1 + exp(-x)).

hard_sigmoid

keras.activations.hard_sigmoid(x)

Hard sigmoid activation function.

Faster to compute than sigmoid activation.

Arguments

• x: Input tensor.

Returns

Hard sigmoid activation:

• 0 if x < -2.5
• 1 if x > 2.5
• 0.2 * x + 0.5 if -2.5 <= x <= 2.5.

exponential

keras.activations.exponential(x)

Exponential (base e) activation function.

Arguments

• x: Input tensor.

Returns

Exponential activation: exp(x).

linear

keras.activations.linear(x)

Linear (i.e. identity) activation function.

Arguments

• x: Input tensor.

Returns

Input tensor, unchanged.

On "Advanced Activations"

Activations that are more complex than a simple TensorFlow/Theano/CNTK function (eg. learnable activations, which maintain a state) are available as Advanced Activation layers, and can be found in the module keras.layers.advanced_activations. These include PReLU and LeakyReLU.