Condition to guarantee $f=0$ on $[a,b]$

Define $g(x) := \int_a^x |f(t)| dt$ for $x \in [a, b]$. Then $g$ is differentiable, non-negative, $g(a) = 0$ and $$ g'(x) = |f(x)| \leq M \int_a^x |f(t)| dt = M g(x) \, . $$ Now let $h(x) := g(x)e^{-Mx}$. Then $h$ is non-negative, $h(a) = 0$ and $$ h'(x) = g'(x) e^{-Mx} - Mg(x) e^{-Mx} \le 0 \, . $$ So $h$ is decreasing on $[a, b]$ and therefore identical to zero.

It follows that $g$ is identically zero, and therefore $|f(x)| = g'(x)$ is also zero on the interval.

(The idea is that $g$ satisfies a "differential inequality" $g' \le Mg$, and to compare it with solutions of the corresponding differential equality $y' = My$, which of course are $y(x) = C e^{Mx}$.)

This is a special case of Grönwall's inequality, an essential tool for any analyst. See the "Integral form for continuous functions" section at the above Wikipedia link, and apply it with $\alpha(t) = 0$ and $\beta(t) = M$. You conclude $f(x) \le 0$ everywhere. Then apply it again to $-f$.

The Wikipedia page also gives the proof, which is pretty much the "integrating factor" argument given by Martin R's answer.