How to make Advanced Activations Layers in Mathematica?
The documentation of ElementwiseLayer
explicitly lists which functions are allowed, and UnitStep
is not in this list, I believe that's why the function fails.
Simple workaround: Use a combination of functions from that list, like Ramp
:
f = Ramp[#] - Ramp[-#]*0.3 &;
l = ElementwiseLayer[f]
l[{-1, -0.5, 0, 0.5, 1}]
{-0.3, -0.15, 0., 0.5, 1.}
This is how to constract a PReLU
data = Thread[RandomReal[1, {100, 2}] -> RandomReal[1, {100, 3}]];
net = NetGraph[{5, ConstantArrayLayer["Output" -> 1],
ReplicateLayer[5], FlattenLayer[],
ThreadingLayer[Ramp[#1] - #2*Ramp[-#1] &], 3},
{NetPort["Input"] -> 1 -> 5 -> 6, 2 -> 3 -> 4 -> 5}]
NetTrain[net, data, MaxTrainingRounds -> 100, BatchSize -> 32];
(*well done*)
In 11.2,mma provide more Advanced Activations Layers