Show that $\int_0^\pi\frac{2\cos(2\theta)+\cos(3\theta)}{5+4\cos(\theta)}d\theta=\frac{\pi}{8}$

If you wish to exploit the residue theorem, then first exploit the fact that the integral is even. In addition, use Euler's formula to write $2\cos(2\theta)+\cos(3\theta)=\text{Re}(2e^{i2\theta}+e^{i3\theta})$.

Then, we have

$$\begin{align} \int_0^\pi \frac{2\cos(2\theta)+\cos(3\theta)}{5+4\cos(\theta)}\,d\theta&=\frac12\text{Re}\left(\oint_{|z|=1}\frac{2z^2+z^3}{5+2(z+z^{-1})}\,\frac{1}{iz}\,dz\right)\\\\ &=\frac12\text{Re}\left(\frac1i\oint_{|z|=1}\frac{z^2(2+z)}{(2z+1)(z+2)}\,dz\right)\\\\ &=\frac12\text{Re}\left(\frac1i\oint_{|z|=1}\frac{z^2}{2z+1}\,dz\right)\\\\ &=\frac{\pi}{8} \end{align}$$

And we are done!

HINT: show that your integrand is equal to $${\frac {4\, \left( \cos \left( x \right) \right) ^{2}-2+4\, \left( \cos \left( x \right) \right) ^{3}-3\,\cos \left( x \right) }{5+4\, \cos \left( x \right) }} $$ and then use the Weierstrass substitution