What is the derivative of ReLU?

5 thoughts on “What is the derivative of ReLU?”

  1. Wouldn’t the derivative be 0 when x is = 0. I see you have a slope, and the derivative keeps on getting larger and larger. I.e. when x is 3, you have the derivative at 3. Shouldn’t it just be a flat line i.e. when x >= 0 { (0,1), (1,1), (2,1)} & when x < 0 {(-1,0), (-2, 0) … (-x, 0)}

    1. Hi Brendan,
      The bottom coloured plot I showed is confusing, and should probably be updated. You are correct that the derivative should be a flat line,where y=1 when x > 0, and y=0 when x<=0. That plot is showing f(x), but the colours are showing f'(x). So the green means f'(x) = 1, and the blue means f'(x) = 0. Hope that helps to clarify.

Questions/comments? If you just want to say thanks, consider sharing this article or following me on Twitter!