Use real integral to calculate $\int_R \frac{x^2 \cos (\pi x)}{(x^2 + 1)(x^2 + 2)}dx$

We can consider a more general function

$$I(t)=2\int_0^\infty\frac{x^2\cos(tx)}{(x^2+1)(x^2+2)}\,\mathrm dx$$

and take its Laplace transform,

$$\begin{align*} \mathcal{L}\left\{I(t)\right\} &= 2s\int_0^\infty\frac{x^2}{(x^2+s^2)(x^2+1)(x^2+2)}\,\mathrm dx\\ &= 2s\int_0^\infty\left( \frac{1}{(1-s^2)(x^2+1)}+\frac{2}{(s^2-2)(x^2+2)}-\frac{s^2}{(s^4-3s^2+2)(x^2+s^2)}\right )\,\mathrm dx\\ &= \frac{\pi\sqrt{2}}{s+\sqrt{2}}-\frac{\pi}{s+1},\qquad\text{do a bunch of arctan integrals and simplify.} \end{align*}$$

 

Now if we use the fact that $\mathcal{L}\left\{e^{-\alpha t}\right\}=\frac{1}{s+\alpha}$, we find

$$I(t)=\pi\sqrt{2}e^{-\sqrt{2}t}-\pi e^{-t}.$$

Plugging in $t=\pi$ gives you what you're looking for.

If you attempt to use the differentiation under the integral sign as you mentioned, you will run into some issues as I highlight below: $$I=\int_{-\infty}^\infty\frac{x^2\cos(\pi x)}{(x^2+1)(x^2+2)}dx=2\int_0^\infty\left(\frac{2\cos(\pi x)}{x^2+2}-\frac{\cos(\pi x)}{x^2+1}\right)dx$$ $$I_1=\int_0^\infty\frac{2\cos(\pi x)}{x^2+2}dx\,\to\,I_1(t)=2\int_0^\infty\frac{\cos(tx)}{x^2+2}dx\tag{1}$$ By differentiating under the integral sign twice we obtain: $$I_1''(t)=-2\int_0^\infty\cos(tx)dx+4\int_0^\infty\frac{\cos(tx)}{x^2+2}dx\tag{2}$$ $$I_1''(t)-2I_1(t)=-2\int_0^\infty\cos(tx)dx\tag{3}$$ Normally by solving this differential equation and applying initial conditions we can get a formula for $I_1(t)$ however this integral on the right does not converge on a value, it instead oscillates so we can say it has an upper and lower bound, but no exact value. This same process could be repeated for an $I_2$ and then let $I=I_1(\pi)-I_2(\pi)$.

This method works well but only in certain situations