Why using fewer terms of Taylor series doesn't give $0/0$ but gives a wrong answer?

You are dividing by $x^3$ at the end, so you need all possible terms at least of degree $3$ in the numerator to be present, otherwise you're basically guaranteed to change the value of the limit.

Let's keep the error term in the third way, and see what happens. I will do that the following way: we have $$ \sin(2x) = 2x + x^3\cdot g(x)\\ \cos(x) = 1-\frac{x^2}2 + x^4\cdot h(x) $$ for some functions $g$ and $h$ where $g(x)$ and $h(x)$ are bounded as $x\to 0$. (It is more common to use $O(x^3)$ instead of $x^3\cdot g(x)$ and $O(x^4)$ rather than $x^4\cdot h(x)$. But the $O$ terms can be a bit unintuitive to arithmetize with, so if you are unaccustomed to working with error terms, I think that my approach here is closer to what you are already used to.)

Then we follow the steps in your third way and see what we get: $$ \frac{2 x \cos x- \sin 2x}{x^3} =\frac{2 x \cos x-(2x + x^3\cdot g(x))}{x^3}\\ =\frac{2x(\cos x -1) - x^3\cdot g(x)}{x^3}\\ =\frac{2x(-\frac{x^2}{2} + x^4\cdot h(x)) + x^3\cdot g(x)}{x^3}\\ =-1 + x\cdot h(x) - g(x) $$ and we see that in order to assess the limit as $x\to 0$, we don't need to know more about $h$ (it is bounded, so $x\cdot h(x)\to 0$), but we do need to know more about $g(x)$. Of course, it is easy to go back and check that $g(x) = -\frac8{3!} + x^2\cdot g_1(x)$ for some function $g_1$ that is bounded for $x\to 0$. Which is enough to conclude that the limit is indeed $\frac13$.


In all the cases we should use remainder to proceed properly as follows

$$\frac{2 x \cos x- \sin 2x}{x^3}=\frac{2 x \left(1-\frac12 x^2+O(x^3)\right)- \left(2x-\frac16 (2x)^3+O(x^4)\right)}{x^3}=$$

$$=\frac{2x-x^3-2x+\frac43x^3+O(x^4)}{x^3}=\frac13+O(x) \to \frac13$$

without remainder we can easily get wrong with the solution.