A definite integral with trigonometric functions: $\int_{0}^{\pi/2} x^{2} \sqrt{\tan x} \sin(2x) \, \mathrm{d}x$

The general method of attack on an integral like this is to recognize that the $x^2$ could be ignored for the time being upon expressing the integral in terms of a suitable parameter which may then be twice differentiated. To wit, after a little manipulatin, I find:

$$\displaystyle \int_{0}^{\frac{\pi}{2}} dx \: {x}^2 \sqrt{\tan x} \sin \left( 2x \right) = 2 \Im{ \int_{0}^{\frac{\pi}{2}} dx \: x^2 e^{i x} \sqrt{\sin{x} \cos{x}}} $$

So consider the following integral:

$$J(\alpha) = \int_{0}^{\frac{\pi}{2}} dx \: e^{i \alpha x} \sqrt{\sin{x} \cos{x}} $$

Note that the integral we seek is

$$\displaystyle \int_{0}^{\frac{\pi}{2}} dx \: {x}^2 \sqrt{\tan x} \sin \left( 2x \right) = -2 \Im{\left [ \frac{\partial^2}{\partial \alpha^2} J(\alpha) \right ]_{\alpha = 1}} $$

It turns out that there is, in fact, a closed for for $J(\alpha)$:

$$J(\alpha) = -\frac{\left(\frac{1}{16}+\frac{i}{16}\right) \sqrt{\frac{\pi }{2}} \left ( e^{\frac{i \pi a}{2}}-i\right) \Gamma \left(\frac{a-1}{4}\right)}{\Gamma \left ( \frac{a+5}{4} \right )} $$

Plugging this into the above expression, you will find terms including polylogs and harmonic numbers, which I will spare you unless explicitly asked for. But this is how you would get a closed-form expression for your integral.

EDIT

The integral is even nicer when we consider

$$J(\alpha) = \int_{0}^{\frac{\pi}{2}} dx \: \sin{\alpha x} \sqrt{\sin{x} \cos{x}} $$

Then

$$J(\alpha) = \frac{\pi ^{3/2} \sin \left(\frac{\pi a}{4}\right)}{8 \Gamma \left(\frac{5-a}{4}\right) \Gamma \left(\frac{5+a}{4}\right)}$$

and

$$\left [ \frac{\partial^2}{\partial \alpha^2} J(\alpha) \right ]_{\alpha = 1} = \frac{\pi \left(-5 \pi ^2+6 \pi (\log (4)-2)+3 (8+(\log (4)-4) \log (4))\right)}{192 \sqrt{2}}$$

The integral we seek is then

$$\int_0^{\frac{\pi}{2}} dx \: x^2 \sqrt{\tan{x}} \sin{(2 x)} = -2 \left [ \frac{\partial^2}{\partial \alpha^2} J(\alpha) \right ]_{\alpha = 1} = \frac{\pi \left(-24+12 \pi +5 \pi ^2-3 \log ^2(4)+12 \log (4)-6 \pi \log (4)\right)}{96 \sqrt{2}}$$

It turns out that the numerical value of the latter value is about $1.10577$, which agrees with the numerical approximation of the integral mentioned by @mrf and verified in Mathematica.

This integral appears as an exercise on page 857 of the textbook Classical Complex Analysis: A Geometric Approach, Volume 1 by I-Hsiung Lin.

You're told to integrate the function $$ f(z) = \log^{2}(z) \Big( \frac{1-z}{1+z} \Big)^{1/2}$$ around an closed indented contour that runs just above the real axis from $z=-1$ to $z=1$ and then along the upper half of the unit circle.

Since the contributions from the indentations around the branch points at $z=0, z=1$, and $z=-1$ can be shown to be vanishing small, we get (using the principal branch of the logarithm throughout),

$$ \int_{-1}^{0} \left(\log|x| + i \pi\right)^{2}\left( \frac{1-x}{1+x} \right)^{1/2} \, dx + \int_{0}^{1} \log^{2}(x) \left(\frac{1-x}{1+x} \right)^{1/2} \, dx $$

$$ \ + i \int_{0}^{\pi} \log^{2}(e^{it}) \left(\frac{1-e^{i t}}{1+e^{i t}} \right)^{1/2} e^{it} \, dt = 0$$

Looking at each piece separately,

$$ \begin{align} \int_{-1}^{0} \left(\log|x| + i \pi \right)^{2} \left( \frac{1-x}{1+x} \right)^{\frac{1}{2}} \, dx &= \int_{0}^{1} \left(\log x + i \pi \right)^{2} \left(\frac{1+x}{1-x} \right)^{1/2} \, dx \\ &= \int_{0}^{1}\left(\log^{2} (x) + 2 \pi i \log x - \pi^{2} \right) \frac{1+x}{\sqrt{1-x^{2}}} \, dx, \end{align}$$

$$\int_{0}^{1} \log^{2}(x) \left(\frac{1-x}{1+x} \right)^{1/2} \ dx = \int_{0}^{1} \log^{2}(x) \frac{1-x}{\sqrt{1-x^{2}}} \, dx, $$

and

$$ \begin{align} i \int_{0}^{\pi} \log^{2}(e^{it}) \Big(\frac{1-e^{i t}}{1+e^{i t}} \Big)^{1/2} e^{it} \ dt &= -i \int_{0}^{\pi} t^{2} \sqrt{-i \tan t/2} \ e^{it} \ dt \\ &= - i e^{-i \pi/4} \int_{0}^{\pi} t^{2} \sqrt{\tan t/2} \ e^{it} \ dt \end{align}$$

So we have

$$ e^{-i \pi /4} \int_{0}^{\pi} t^{2} \sqrt{\tan t/2} \, e^{it} \, dt = -2i \int_{0}^{1} \frac{\log^{2}(x)}{\sqrt{1-x^{2}}} \, dx + 2 \pi \int_{0}^{1} \frac{\log x}{\sqrt{1-x^{2}}} \, dx $$

$$ + 2 \pi \int_{0}^{1} \frac{x \log x}{\sqrt{1-x^{2}}} \, dx +i \pi^{2} \int_{0}^{1} \frac{1}{\sqrt{1-x^{2}}} \, dx +i \pi^{2} \int_{0}^{1} \frac{x}{\sqrt{1-x^{2}}} \, dx$$

We can use the fact that

$$ \int_{0}^{1} \frac{x^{a}}{\sqrt{1-x^{2}}} \ dx = \frac{1}{2} \int_{0}^{1} u^{a/2+1/2-1} (1-u)^{1/2-1} \ du = \frac{1}{2} B \left( \frac{1}{2}, \frac{a}{2}+ \frac{1}{2} \right)$$

along with properties of the beta function (entries 23-26) to evaluate the first three integrals on the right side of the equation.

$$ \begin{align} \int_{0}^{1} \frac{\log^{2}(x)}{\sqrt{1-x^{2}}} \, dx &= \frac{d^{2}}{da^{2}} \frac{1}{2} B \left( \frac{1}{2}, \frac{a}{2}+ \frac{1}{2} \right) \Bigg|_{a=0} \\ &= \frac{1}{8} B \left( \frac{1}{2}, \frac{1}{2} \right) \left[ \left( \psi \left(\frac{1}{2} \right) - \psi(1) \right)^{2} + \psi_{1}\left(\frac{1}{2}\right) - \psi_{1}(1) \right] \\ &= \frac{\pi^{3}}{24} + \frac{\pi \log^{2} 4}{8} \end{align}$$

$$ \begin{align} \int_{0}^{1} \frac{\log x}{\sqrt{1-x^{2}}} \, dx &= \frac{d}{da} \frac{1}{2} B \left( \frac{1}{2}, \frac{a}{2}+ \frac{1}{2} \right) \Bigg|_{a=0} \\ &= \frac{1}{4} B \left( \frac{1}{2}, \frac{1}{2} \right) \left( \psi\left(\frac{1}{2}\right) - \psi(1) \right) \\ &= - \frac{\pi \log 2}{2} \end{align}$$

$$ \begin{align} \int_{0}^{1} \frac{x \log x}{\sqrt{1-x^{2}}} \, dx &= \frac{d}{da} \frac{1}{2} B \left( \frac{1}{2}, \frac{a}{2}+ \frac{1}{2} \right) \Bigg|_{a=1} \\ &= \frac{1}{4} B \left( \frac{1}{2}, 1 \right) \left( \psi(1) - \psi\left(\frac{3}{2}\right) \right) \\ &= \log 2 -1 \end{align}$$

Therefore,

$$ \int_{0}^{\pi} t^{2} \sqrt{\tan t/2} \ e^{it} \ dt = e^{i \pi /4} \left( -\frac{i \pi^{3}}{12} - \frac{i\pi \log^{2} 4}{4} - \pi^{2} \log 2 + 2 \pi \log 2 - 2 \pi + \frac{i\pi^{3}}{2} +i \pi^{2} \right)$$

Now just make the substitution $u = \frac{t}{2}$, and then equate the imaginary parts on both sides of the equation.