Evaluation of $\int_0^{\pi/4} \sqrt{\tan x} \sqrt{1-\tan x}\,\,dx$

\begin{align} \int_0^{\Large\frac{\pi}{4}} \sqrt{\tan x} \sqrt{1-\tan x}\,\,dx&=\int_0^1\frac{\sqrt{y(1-y)}}{1+y^2}\,dy\quad\Rightarrow\quad y=\tan x\\ &=\int_0^\infty\frac{\sqrt{t}}{(1+t)(1+2t+2t^2)}\,dt\quad\Rightarrow\quad t=\frac{y}{1-y}\\ &=\int_0^\infty\frac{2z^2}{(1+z^2)(1+2z^2+2z^4)}\,dz\quad\Rightarrow\quad z^2=t\\ &=2\int_0^\infty\left[\frac{2z^2}{1+2z^2+2z^4}+\frac{1}{1+2z^2+2z^4}-\frac{1}{1+z^2}\right]\,dz\\ &=\int_{-\infty}^\infty\left[\frac{2z^2}{1+2z^2+2z^4}+\frac{1}{1+2z^2+2z^4}-\frac{1}{1+z^2}\right]\,dz\\ &=I_1+I_2-\pi \end{align}

---

\begin{align} I_1 &=\int_{-\infty}^\infty\frac{2z^2}{1+2z^2+2z^4}\,dz\\ &=\int_{-\infty}^\infty\frac{1}{z^2+\frac{1}{2z^2}+1}\,dz\\ &=\int_{-\infty}^\infty\frac{1}{\left(z-\frac{1}{\sqrt{2}z}\right)^2+1+\sqrt{2}}\,dz\\ &=\int_{-\infty}^\infty\frac{1}{z^2+1+\sqrt{2}}\,dz\\ &=\frac{\pi}{\sqrt{1+\sqrt{2}}} \end{align} where the 4th line we use identity

\begin{align} \int_{-\infty}^\infty f\left(x\right)\,dx=\int_{-\infty}^\infty f\left(x-\frac{a}{x}\right)\,dx\qquad,\qquad\text{for }\, a>0. \end{align}

The proof can be seen in my answer here. $I_2$ can be proved in similar manner (see user111187's answer). \begin{equation} I_2=\frac{1}{2}\int_{-\infty}^\infty\frac{1}{z^4+z^2+\frac{1}{2}}\,dz=\pi\sqrt{\frac{\sqrt{2}-1}{2}} \end{equation}

---

Combine all the results together, we finally get

\begin{equation} \int_0^{\Large\frac{\pi}{4}} \sqrt{\tan x} \sqrt{1-\tan x}\,\,dx=\frac{\pi}{\sqrt[4]{2}}\sqrt{\frac{2+\sqrt{2}}{2}}-\pi \end{equation}

Here is a complex analysis approach. Integrate $$f(z)=\frac{\sqrt{z}\sqrt{z-1}}{z^2+1}=\frac{|z|^\frac{1}{2}|z-1|^\frac{1}{2}e^{i\varphi}}{z^2+1}$$ where $\varphi=\dfrac{1}{2}\left(\arg{z}+\arg(z-1)\right)$, $0\le \arg{z}, \arg(z-1)\le 2\pi$, along a dumbbell contour. Just above $[0,1]$, $\varphi=\dfrac{\pi}{2}$, and just below $[0,1]$, $\varphi=\dfrac{3\pi}{2}$. So the contour integral is \begin{align} 2i\int^1_0\frac{\sqrt{x}\sqrt{1-x}}{1+x^2}{\rm d}x =&2\pi i\left[\operatorname*{Res}_{z=i}f(z)+\operatorname*{Res}_{z=-i}f(z)-\operatorname*{Res}_{z=0}\frac{f(z^{-1})}{z^2}\right]\\ =&2\pi i\left[\frac{\sqrt[4]{2}}{2i}e^{i5\pi/8}-\frac{\sqrt[4]{2}}{2i}e^{i11\pi/8}-1\right]\\ =&2\pi i\left[\frac{\sqrt[4]{2}}{2i}\left(e^{i3\pi/8}-e^{-i3\pi/8}\right)-1\right]\\ =&2\pi i\left[\sqrt[4]{2}\sin\left(\frac{3\pi}{8}\right)-1\right] \end{align} Therefore $$\int^\frac{\pi}{4}_0\sqrt{\tan{x}-\tan^2{x}}\ {\rm d}x=\int^1_0\frac{\sqrt{x}\sqrt{1-x}}{1+x^2}{\rm d}x=\pi\left[\sqrt[4]{2}\sin\left(\frac{3\pi}{8}\right)-1\right]$$

To continue the work of Anastasiya-Romanova but not using complex analysis

For $I_1$:

Notice: $p(z)=1+2z^2+2z^4=2\Big(z^2+az+b\Big)\Big(z^2-az+b\Big)$

where $a=\sqrt{\sqrt{2}-1}$ and $b=\dfrac{\sqrt{2}}{2}$

Therefore:

$f(z)=\displaystyle \dfrac{2z^2}{1+2z^2+2z^4}=\dfrac{z}{2a\Big(z^2-az+b\Big)}-\dfrac{z}{2a\Big(z^2+az+b\Big)}$ $f(z)=\dfrac{2z-a}{4a\Big(z^2-az+b\Big)}-\dfrac{2z+a}{4a\Big(z^2+az+b\Big)}+\dfrac{1}{4\Big(z^2-az+b\Big)}+\dfrac{1}{4\Big(z^2+az+b\Big)}$

Let $c=b-\dfrac{a^2}{4}$, $c>0$

Therefore:

$f(z)=\dfrac{2z-a}{4a\Big(z^2-az+b\Big)}-\dfrac{2z+a}{4a\Big(z^2+az+b\Big)}+\dfrac{1}{4\Big(\big(z-\tfrac{a}{2}\big)^2+c\Big)}+\dfrac{1}{4\Big(\big(z+\tfrac{a}{2}\big)^2+c\Big)}$

So a primitive of $\displaystyle \dfrac{2z}{1+2z^2+2z^4}$ is:

$\dfrac{1}{4a}\log\Big(\dfrac{z^2-az+b}{z^2+az+b}\Big)+\dfrac{1}{4\sqrt{c}}\arctan\Big(\dfrac{z-\tfrac{a}{2}}{\sqrt{c}}\Big)+\dfrac{1}{4\sqrt{c}}\arctan\Big(\dfrac{z+\tfrac{a}{2}}{\sqrt{c}}\Big)$

(think about derivative of $\log(u(x))$ )

Therefore:

$\displaystyle \int_{-\infty}^{+\infty}\dfrac{2x^2dx}{1+2x^2+2x^4}=\dfrac{\pi}{2\sqrt{c}}=\pi\sqrt{\sqrt{2}-1}$

To compute $I_2$ start performing change of variable $u=\dfrac{1}{x}$ , the function to integrate becomes $\dfrac{x^2}{x^4p\Big(\dfrac{1}{x}\Big)}$

$q(x)=x^4p\Big(\dfrac{1}{x}\Big)$

$q(x)=2x^4\Big(\dfrac{1}{x^2}+\dfrac{a}{x}+b\Big)\Big(\dfrac{1}{x^2}-\dfrac{a}{x}+b\Big)$

$q(x)=2(1+ax+bx^2)(1-ax+bx^2)$

$q(x)=2b^2\Big(x^2+\dfrac{a}{b}x+\dfrac{1}{b}\Big)\Big(x^2-\dfrac{a}{b}x+\dfrac{1}{b}\Big)$

The new $a,b$ are respectively $\dfrac{a}{b},\dfrac{1}{b}$ and there is a new $c$.

Therefore:

$\displaystyle \int_{-\infty}^{+\infty}\dfrac{dx}{1+2x^2+2x^4}=\dfrac{2}{b^2}\times \dfrac{\pi}{2\sqrt{c}}=\dfrac{\pi}{b^2\sqrt{c}}=\pi\dfrac{\sqrt{2}}{2}\sqrt{\sqrt{2}-1}$