$\lim_{x\to 0}\frac{\sin 3x+A\sin 2x+B\sin x}{x^5}$ without series expansion or L Hospital rule

Using trigonometric identities we have \begin{align} 3-4\sin^2 x+2A\cos x+B&=4(1-\sin^2 x)+2A\cos\left(2\cdot\frac{x}{2}\right)+B-1\\ &=4\cos^2\left(2\cdot\frac{x}{2}\right)+2A\cos\left(2\cdot\frac{x}{2}\right)+B-1\\ &=4\left(1-2\sin^2\frac{x}{2}\right)^2+2A\left(1-2\sin^2 \frac{x}{2}\right)+B-1\\ &=16\sin^4\frac{x}{2}-16\sin^2\frac{x}{2}-4A\sin^2\frac{x}{2}+2A+B+3\\ &=16\sin^4\frac{x}{2}-4(A+4)\sin^2\frac{x}{2}+2A+B+3\\ \end{align} In order to make the limit finite we must have $$A+4=0\quad\text{and}\quad 2A+B+3=0\qquad\iff\qquad \color{blue}{A=-4}\quad\text{and}\quad \color{blue}{B=5}$$ By taking those values we get \begin{align} \lim_{x\to 0}\frac{\sin 3x\color{blue}{-4}\sin 2x+\color{blue}{5}\sin x}{x^5}&=\left(\lim_{x\to 0}\frac{16\sin^4\frac{x}{2}}{x^4}\right)\left(\lim_{x\to 0}\frac{\sin x}{x}\right)\\ &=\left(\lim_{x\to 0}\frac{\sin \frac{x}{2}}{\frac{x}{2}}\right)^4\left(1\right)\\ &=\color{blue}{1} \end{align} Since $f$ is continuous at $0$ it follows $f(0)=1$.

Your first step is correct, you need $3+2A+B=0$ for the function to converge. Let's substitute $B=-3-2A$ into the original function. Them you have

$$ f(x)=\frac{\sin(3x)+A\sin(2x)+(-3-2A)\sin(x)}{x^5} $$

Using the angle sum formulas: $$ \sin(3x)=\sin(x)\cos(2x)+\cos(x)\sin(2x)=\sin(x)(\cos^2(x)-\sin^2(x))+2\sin(x)\cos(x)^2. $$ Now, using the standard trig identity, $\sin^2(x)=1-\cos^2(x)$. Therefore, $$ \sin(3x)=4\sin(x)\cos(x)^2-\sin(x) $$

Also using the angle sum formulas: $$ \sin(2x)=2\sin(x)\cos(x) $$

Substituting these into the original function, we have $$ f(x)=\frac{4\sin(x)\cos^2(x)-\sin(x)+2A\sin(x)\cos(x)-3\sin(x)-2A\sin(x)}{x^5}. $$ By combining the terms with $A$ and those without, we have $$ f(x)=\frac{4\sin(x)(\cos^2(x)-1)+2A\sin(x)(\cos(x)-1)}{x^5}=\frac{\sin(x)}{x}\cdot\frac{4(\cos^2(x)-1)+2A(\cos(x)-1)}{x^4} $$ Factoring a difference of squares, the numerator becomes $$ f(x)=\frac{\sin(x)}{x}\cdot\frac{4(\cos(x)-1)(\cos(x)+1)+2A(\cos(x)-1)}{x^4}$$ Factoring again, we have $$f(x)=\frac{\sin(x)}{x}\cdot\frac{\cos(x)-1}{x^2}\cdot\frac{4(\cos(x)+1)+2A}{x^2} $$

Observe that $$ \lim_{x\rightarrow 0}\frac{\cos(x)-1}{x^2}=\lim_{x\rightarrow 0}\frac{\cos(x)-1}{x^2}\cdot\frac{\cos(x)+1}{\cos(x)+1}=\lim_{x\rightarrow 0}\frac{\cos^2(x)-1}{x^2(\cos(x)+1)} $$ Then, using the standard trig identities, we get that this limit equals $$ \lim_{x\rightarrow 0}\frac{-\sin^2(x)}{x^2(\cos(x)+1)}=\lim_{x\rightarrow 0}\left(\frac{\sin(x)}{x}\right)^2\frac{-1}{\cos(x)+1}, $$ which we can see has a limit of $-\frac{1}{2}$.

Continuing from here, you also need the numerator to vanish for the limit to exist because the denominator vanishes. In other words, $\lim_{x\rightarrow 0}4(\cos(x)+1)+2A=8+2A$ must be $0$, so $A=-4$. After substitution, the entire formula becomes $$f(x)=\frac{\sin(x)}{x}\cdot\frac{\cos(x)-1}{x^2}\cdot\frac{4(\cos(x)-1)}{x^2} $$ Substituting the values that we've calculated the limit as $$ \lim_{x\rightarrow 0}f(x)=1\cdot\left(-\frac{1}{2}\right)\cdot\left(-\frac{4}{2}\right)=1. $$