How do you solve $\int_{0}^{\frac{\pi}{2}} \frac{2304\cos x}{(\cos 4x-8\cos 2x+15)^2} \,dx$

Here are a few shortcuts. With $t =\sin x$

\begin{equation} I=\int_{0}^{\frac{\pi}{2}} \frac{2304\cos x}{(\cos 4x-8\cos 2x+15)^2} \,dx=36\int_0^1 \frac {1}{(t^4+t^2+1)^2}dt\tag1 \end{equation} Note $$\left(\frac{t-t^3}{t^4+t^2+1}\right)’ = \frac{t^6+5}{(t^4+t^2+1)^2}-\frac{4}{t^4+t^2+1} $$ Integrate both sides over $(0,\infty)$

\begin{align} \int_0^\infty \frac{4}{t^4+t^2+1}dt &= \int_0^\infty \overset{t\to 1/t} {\frac{t^6}{(t^4+t^2+1)^2} }dt + \int_0^\infty \frac{5}{(t^4+t^2+1)^2}dt\\ &= \int_0^1\frac{6}{(t^4+t^2+1)^2}dt +\int_1^\infty \frac{6}{(t^4+t^2+1)^2}dt\tag2 \end{align}

Similarly, integrate over $(0,1)$

\begin{align} \int_0^1\frac{4}{t^4+t^2+1}dt=\int_0^1\frac{5}{(t^4+t^2+1)^2}dt +\int_1^\infty \frac{1}{(t^4+t^2+1)^2}dt \tag3 \end{align} Combine (2) and (3) to obtain \begin{align} \int_0^1\frac{dt}{(t^4+t^2+1)^2}= \int_0^1\frac{dt}{t^4+t^2+1} -\frac14 \int_0^\infty \frac{dt}{t^4+t^2+1}= \frac14\ln3+\frac{\pi}{6\sqrt3} \end{align} Plug into (1) $$I= 9\ln3+ 2\sqrt3{\pi}$$

---

P.S. The last integral is carried out as follows

$$\int \frac{1}{t^4+t^2+1}dt= \frac12 \int\frac{1-t^2}{t^4+t^2+1}dt + \frac12 \int\frac{1+t^2}{t^4+t^2+1}dt \\ = \frac12 \int\frac{d(t+\frac1t)}{(t+\frac1t)^2-1} + \frac12 \int\frac{d(t-\frac1t)}{(t-\frac1t)^2+3} $$

Tou have done a good work and obtained the correct result.

Trying on my side, using as you did $s=\sin(x)$, we ned with $$\int_{0}^{\frac{\pi}{2}} \frac{2304\cos x}{(\cos 4x-8\cos 2x+15)^2} \,dx=36 \int_{0}^1 \frac{ds}{ \left(s^4+s^2+1\right)^2}$$

$$s^4+s^2+1=\left(s^2-s+1\right) \left(s^2+s+1\right)$$ Using partial fraction decomposition $$\frac{1}{ \left(s^4+s^2+1\right)^2}=\frac{1-s}{2 \left(s^2-s+1\right)}+\frac{s+1}{2 \left(s^2+s+1\right)}-$$ $$\frac{s}{4 \left(s^2-s+1\right)^2}+\frac{s}{4 \left(s^2+s+1\right)^2}$$ The first and the second are not difficult $$I_1=\int\frac{1-s}{ s^2-s+1}\,ds=-\frac 12\Bigg[\int \frac{2s-1}{ s^2-s+1}ds -\int\frac{ds}{ s^2-s+1}\Bigg]$$ $$I_2=\int\frac{s+1}{s^2+s+1}ds=\frac 12\Bigg[\int \frac{2s+1}{ s^2+s+1}ds -\int\frac{ds}{ s^2+s+1}\Bigg]$$ Now, for the third and fourth antiderivatives which look like $$J=\int \frac s{(s^2+as+1)^2} ds=\int \frac s{(s-r_1)^2(s-r_2)^2} ds$$ partial fraction decomposition again $$\frac s{(s-r_1)^2(s-r_2)^2}=\frac{r_1}{(r_2-r_1)^2 (s-r_1)^2}+\frac{r_1+r_2}{(r_2-r_1)^3 (s-r_1)}-$$ $$\frac{r_1+r_2}{(r_2-r_1)^3 (s-r_2)}+\frac{r_2}{(r_2-r_1)^2 (s-r_2)^2}$$

This is a pure nightmare !

At the end, after recombining evrything, the antiderivative is $$\frac{6 s\left(1-s^2\right)}{s^4+s^2+1}+4 \sqrt{3} \tan ^{-1}\left(\frac{16 s \left(1-s^2\right)}{3 \sqrt{3}}\right)+9 \log \left(\frac{s^2+s+1}{s^2-s+1}\right)$$ and, for the definite integral, your good result $$2 \sqrt{3} \pi +9 \log (3)$$

All this work took me more than one hour. Be sure that I have been looking for tricks but ... no one came to my mind.