What is the derivative of ReLU?

How to Compute the Derivative of a Sigmoid Function (fully worked example)

This is a sigmoid function:

\boldsymbol{s(x) = \frac{1}{1 + e^{-x}}}  

The sigmoid function looks like this (made with a bit of MATLAB code):

x=-10:0.1:10;
s = 1./(1+exp(-x));
figure; plot(x,s); title('sigmoid');

sigmoid

Alright, now let’s put on our calculus hats…
Continue reading “How to Compute the Derivative of a Sigmoid Function (fully worked example)”