How to implement PReLU activation in Tensorflow?
I think it's much easier to implement with tf.maximum
My implementation is as following
import tensorflow as tf
def PReLU(_x, name=None):
if name is None:
name = "alpha"
_alpha = tf.get_variable(name,
shape=_x.get_shape(),
initializer=tf.constant_initializer(0.0),
dtype=_x.dtype)
return tf.maximum(_alpha*_x, _x)
Although the solution with tf.maximum
is very efficient, it can't represent a concave function. Here's a solution that can:
def prelu(_x, scope=None):
"""parametric ReLU activation"""
with tf.variable_scope(name_or_scope=scope, default_name="prelu"):
_alpha = tf.get_variable("prelu", shape=_x.get_shape()[-1],
dtype=_x.dtype, initializer=tf.constant_initializer(0.1))
return tf.maximum(0.0, _x) + _alpha * tf.minimum(0.0, _x)
The implementation of PReLU seems straight-forward based on the PreLU implementations (see: Keras, TFLearn and TensorLayer) of the higher level libraries. My code is as follows:
def parametric_relu(_x):
alphas = tf.get_variable('alpha', _x.get_shape()[-1],
initializer=tf.constant_initializer(0.0),
dtype=tf.float32)
pos = tf.nn.relu(_x)
neg = alphas * (_x - abs(_x)) * 0.5
return pos + neg