Find $(a,b)$ such that $\lim\limits_{x\to 0}\frac{f(x)}{x}=1$ implies $\lim\limits_{x\to 0}\frac{x(1+a\cos x)-b\sin x}{(f(x))^3}=1$

Since we can write $$\lim_{x\to 0}\frac{x(1+a\cos x)-b\sin x}{(f(x))^3}=\lim_{x\to 0}\frac{x(1+a\cos x)-b\sin x}{x^3}\cdot\frac{1}{(f(x)/x)^3}$$ we have to have $$\lim_{x\to 0}\frac{x(1+a\cos x)-b\sin x}{x^3}=1$$ We can write $$\lim_{x\to 0}\frac{x(1+a\cos x)-b\sin x}{x^3}\tag1$$$$=\lim_{x\to 0}\frac{1+a\cos x-b\frac{\sin x}{x}}{x^2}\tag2$$ But we cannot write $(2)$ as $$\lim_{x\to 0}\frac{1+a\cos x-b\cdot 1}{x^2}$$

---

From $(1)$, by L'Hôpital's rule, $$(1)=\lim_{x\to 0}\frac{1+a\cos x-ax\sin x-b\cos x}{3x^2}\tag3$$ Here, we have to have $$1+a-b=0\tag 4$$ Using L'Hôpital's rule several times, $$\begin{align}(3)&=\lim_{x\to 0}\frac{1-ax\sin x-\cos x}{3x^2}\\&=\lim_{x\to 0}\frac{-a(\sin x+x\cos x)+\sin x}{6x}\\&=\lim_{x\to 0}\frac{-2a\cos x+ax\sin x+\cos x}{6}\\&=\frac{-2a+1}{6}\end{align}$$ and so $$\frac{-2a+1}{6}=1\tag5$$ Now solve $(4)(5)$.

Just as mathlove answered, we need to find $a,b$ such that $$\lim_{x\to 0}\frac{x(1+a\cos (x))-b\sin (x)}{x^3}=1$$ Let us use the classical Taylor expansions $$\cos(x)=1-\frac{x^2}{2}+O\left(x^3\right)$$ $$\sin(x)=x-\frac{x^3}{6}+O\left(x^4\right)$$ and replace to get $$\frac{x(1+a\cos (x))-b\sin (x)}{x^3}=\frac{x (a-b+1)+x^3 \left(\frac{b}{6}-\frac{a}{2}\right)+O\left(x^4\right)}{x^3}$$ So, the first thing is $a-b+1=0$ and the second (for a limit equal to $1$) is $\frac{b}{6}-\frac{a}{2}=1$. Solving these two equations leads to the required values.