Compute $\int_0^{\pi/2} x^2\left(\sum_{n=1}^\infty (-1)^{n-1} \cos^n(x)\cos(nx)\right)dx$

We can use the following fourier series:$$\frac{1}{a+b\cos t}=\frac{1}{\sqrt{a^2-b^2}}+\frac{2}{\sqrt{a^2-b^2}}\sum_{n=1}^{\infty}\left(\frac{\sqrt{a^2-b^2}-a}{b}\right)^n\cos{(nt)},\ a>b$$ Plugging $a=5, b=3$ and $t=2x$ we get: $$\frac{1}{1+3\cos^2 x}=\frac{2}{5+3\cos(2x)}=\frac{1}{2}+\sum_{n=1}^\infty (-1)^n\left(\frac{1}{3}\right)^n\cos(2nx)$$ $$\Rightarrow \int_0^\frac{\pi}{2}\frac{x^2}{1+3\cos^2 x}dx=\frac12\int_0^\frac{\pi}{2} x^2dx+\sum_{n=1}^\infty(-1)^n \left(\frac13\right)^n\int_0^\frac{\pi}{2}x^2 \cos(2nx)dx$$ $$=\frac{\pi^3}{48}+\frac{\pi}4\sum_{n=1}^\infty \left(\frac13\right)^n\frac{1}{n^2}=\frac{\pi^3}{48}+\frac{\pi}{4}\operatorname{Li}_2\left(\frac13\right)$$

---

Using the series obtain above, we can also conclude that: $$\sum_{n=1}^{\infty}(-1)^n \cos^n(x) \cos(nx)=-\frac13+\frac23\sum_{n=1}^\infty \left(-\frac{1}{3}\right)^n\cos(2nx)$$

Evaluating the blue integral:

First we write

$$\frac1{1+3\cos^2(x)}=\frac{1}{5+3\cos(2x)}$$

Using the same identity in the post body

$$\sum_{n=1}^{\infty}p^n \cos(nx)=\frac{p(\cos(x)-p)}{1-2p\cos(x)+p^2}, \ |p|<1\tag1$$

But lets manipulate the denominator to have it in the form of $\frac1{5+3\cos(x)}$:

$$\frac1{1-2p\cos(x)+p^2}=\frac{-\frac{3}{2p}}{-\frac{3(1+p^2)}{2p}+3\cos(x)}$$

Now set $$-\frac{3(1+p^2)}{2p}=5\Longrightarrow p=-3,-\frac13$$

and since $|p|<1$, so we take $p=-\frac13$. Plug this value in (1) and replace $x$ by $2x$ we get

$$\frac{1}{5+3\cos(2x)}=\frac{1}{4}+\frac12\sum_{n=1}^\infty (-1)^n\left(\frac{1}{3}\right)^n\cos(2nx)\tag2$$

Multiply both sides of (2) by $x^2$ and integrate between $0$ and $\pi/2$ we get

$$\int_0^{\pi/2}\frac{x^2}{1+3\cos^2(x)}dx=\frac{\pi^3}{48}+\frac{\pi}{4}\operatorname{Li}_2\left(\frac13\right)$$

---

Addendum:

The identity used by @Zacky above:

$$\frac{1}{a+b\cos(x)}=\frac{1}{\sqrt{a^2-b^2}}+\frac{2}{\sqrt{a^2-b^2}}\sum_{n=1}^{\infty}\left(\frac{\sqrt{a^2-b^2}-a}{b}\right)^n\cos{(nx)},\ a>b\tag{3}$$

can be derived the same way:

$$\frac1{1-2p\cos(x)+p^2}=\frac{-\frac{b}{2p}}{-\frac{b(1+p^2)}{2p}+b\cos(x)}$$

If we set $$-\frac{b(1+p^2)}{2p}=a\tag{4}$$

we can write

$$\frac1{1-2p\cos(x)+p^2}=\frac{\frac{a}{1+p^2}}{a+b\cos x}$$

We proved above that

$$\sum_{n=1}^{\infty}p^n \cos(nx)=\frac{p(\cos(x)-p)}{1-2p\cos(x)+p^2}=-\frac12-\frac{p^2-1}{1-2p\cos(x)+p^2}$$

$$=-\frac12-\frac12 \color{red}{\frac{p^2-1}{p^2+1}}\frac{\color{red}{a}}{1-2p\cos(x)+p^2}\tag5$$

From $(4)$ we find $p=\frac{\sqrt{a^2-b^2}-a}{b}$. Note that we ignored $p=\frac{\sqrt{a^2-b^2}+a}{b}$ as $|p|<1$.

Substitute this root in $(5)$ we get

$$\sum_{n=1}^\infty\left(\frac{\sqrt{a^2-b^2}-a}{b}\right)^n\cos(nx)=-\frac12-\frac12\cdot\frac{\color{red}{-\sqrt{a^2-b^2}}}{a+b\cos(x)}$$

or

$$\frac{1}{a+b\cos(x)}=\frac{1}{\sqrt{a^2-b^2}}+\frac{2}{\sqrt{a^2-b^2}}\sum_{n=1}^{\infty}\left(\frac{\sqrt{a^2-b^2}-a}{b}\right)^n\cos{(nx)}$$