Improved intermediate value theorem

Counterexample

Let $f(x) = x^2 \sin^2(1/x) \operatorname{sign}(x)$ with $f(0)=0$ (plot). For $x<0$ the function oscillates between $0$ and negative values, and for $x>0$ it oscillates between $0$ and positive values. At $x=0$ the function crosses the $x$-axis, but it has zeros arbitrarily close to $x=0$ so there is no interval $(-\epsilon,0)$ on which $f$ is strictly negative, and no interval $(0, \epsilon)$ on which $f$ is strictly positive.

Do you know the Cantor set? Perhaps you can construct a continuous function $f$ on $[0,1]$ so that:

$f$ is positive on $\left(\frac{1}{3},\frac{2}{3}\right)$
$f$ is negative on $\left(\frac{1}{9},\frac{2}{9}\right)$ and $\left(\frac{7}{9},\frac{8}{9}\right)$
$f$ is positive on $\left(\frac{1}{27},\frac{2}{27}\right)$ and $\left(\frac{7}{27},\frac{8}{27}\right)$ and $\left(\frac{19}{27},\frac{20}{27}\right)$ and $\left(\frac{25}{27},\frac{26}{27}\right)$
and so on, with finally $f$ zero on the rest (the Cantor set).

Then take $a=1/6, b=1/2$.

This is possible if $f$ does not need to vanish on $[s_1,s_2]$ — and impossible if it needs to.

I will first consider the case where $f$ does not necessarily vanish on $[s_1,s_2]$. Let $Z=f^{-1}(0)$ be the set of zeroes of $f$ on $[a,b]$. By the intermediate value theorem $Z\neq\emptyset$.

We can let $s_1=\min Z$ and $s_2=\max Z$. The set $Z$ is a closed (and compact) set, so the minimum and maximum exist. If you are not convinced, replace minimum and maximum by infimum and supremum and show that they must in fact be in $Z$ by continuity of $f$.

We have $a<s_1\leq s_2<b$ since $f(a)\neq0\neq f(b)$. On $[a,s_1)$ the function $f$ only takes negative values. If there was a positive or zero value, the intermediate value theorem would lead to a contradiction with minimality of $s_1$. Similarly $f$ is positive on $(s_2,b]$. If you choose $\epsilon=\frac12\min(s_1-a,b-s_2)$, the desired properties are satisfied apart from $f|_{[s_1,s_2]}\equiv0$.

However, the requirement that $f$ vanishes between $s_1$ and $s_2$ is generally not possible to satisfy for smooth functions. Now we can no longer simply choose $s_1=\min Z$ and $s_2=\max Z$ — and indeed there are functions for which no choice works. For every closed subset $C\subset(a,b)$ there is a smooth function $f\colon[a,b]\to\mathbb R$ so that $f(a)<0<f(b)$ and $f^{-1}(0)=C$. (See this MathOverflow question. I will not go into the details here.)

Now if $C$ is such that all points are accumulation points but the interior is empty, then your desired properties cannot all be satisfied. Since $C$ has empty interior, we necessarily have $s_1=s_2$. (By assumption $[s_1,s_2]\subset C$, so $s_1<s_2$ implies that $C$ has interior points.) The zeroes accumulate at $s_1=s_2\in C$, so for any $\epsilon>0$ there are zeroes in $(s_1-\epsilon,s_1]$ or $[s_2,s_2+\epsilon)$.

On yet another note, your claim is true if $f$ is real analytic. Now there are finitely many zeroes and they are isolated. (The zeroes of a non-constant analytic function cannot accumulate.) Isolation implies $s_1=s_2$. This zero needs to be such that $f$ is negative left of it and positive right of it (near the point).

The zeroes $z_1<z_2<\dots<z_n$ of $f$ split $(a,b)$ into open subintervals $(a,z_1),(z_1,z_2),\dots$. On each of these intervals $f$ is positive or negative. There are two adjacent intervals so that $f<0$ on the left one and $f>0$ on the right one. (Such a zero has odd degree.) Then the zero $z$ between these intervals will work as your $z=s_1=s_2$. You can also explicitly choose $z=\inf\{x\in[a,b];f(x)>0\}$ if you want. If you want more details on this, I recommend asking a follow-up question. The detailed properties of real analytic functions would be too much of a sidetrack here.