ReLU derivative in backpropagation
Relu Derivative is 1 for x >= 0 and 0 for x < 0
The relu derivative can be implemented with np.heaviside step function e.g. np.heaviside(x, 1). The second parameter defines the return value when x = 0, so a 1 means 1 when x = 0.
since ReLU doesn't have a derivative.
No, ReLU has derivative. I assumed you are using ReLU function f(x)=max(0,x)
. It means if x<=0
then f(x)=0
, else f(x)=x
. In the first case, when x<0
so the derivative of f(x) with respect to x gives result f'(x)=0
. In the second case, it's clear to compute f'(x)=1
.