What's wrong with this use of Taylor's expansions?

Hint $$\sin(x\sin(x))=x^2-\frac{x^4}{6}+O\left(x^5\right)$$ would help.

In fact, and this is a general rule, if the denominator is $x^n$, you must develop the numerator at least to order $n$.

Since you have $x^4$ in the denominator, you need to expand the numerator up to $x^4$. You did that for $x^2\cos x$ but not for $\sin (x \sin x)$.

So, you need to include terms up to $x^4$ in the expansion of $\sin (x \sin x)$: $$\sin (x \sin x) = x^2-x^4/6+o(x^6)$$ This comes for $$\sin (x) = x-x^3/6+o(x^5)$$ $$x\sin (x) = x^2-x^4/6+o(x^6)$$ which you then feed to $$\sin (y) = y+o(y^3)$$

I suspect the mistake to be in the expansion of $~\sin(x\sin x),$ but I don't get it. What's wrong ?

Your error consists in using the double-approximation $\sin(x\sin x)\simeq x\sin x\simeq x^2,$ by applying

$\sin t\simeq t$ twice instead of just once, yielding the more accurate $\sin(x\sin x)\simeq x\sin x.$ The latter

leads to $~\lim\limits_{x\to0}~\dfrac{\cos x-\dfrac{\sin x}x}{x^2}=-\dfrac13,~$ which is different from $~\lim\limits_{x\to0}~\dfrac{\cos x-\color{red}1}{x^2}=-\dfrac12.$