Solve integral $\int^1_0 \frac{1-x^2}{{(1+x^2)}\sqrt{1+x^4}}dx$ using subsituition $\sqrt{1+x^4} = {(1+x^2)}\cos{\theta}$

Divide top and bottom by $x^2$:

$$\int_0^1\frac{\frac{1}{x^2}-1}{\left(\frac{1}{x}+x\right)\sqrt{\frac{1}{x^2}+x^2}}dx$$

which suggests using the substitution $t = \frac{1}{x}+x$:

$$\int_2^\infty \frac{dt}{t\sqrt{t^2-2}} = \frac{1}{\sqrt{2}}\sec^{-1}\left(\frac{t}{\sqrt{2}}\right)\Biggr|_2^\infty = \frac{\pi}{4\sqrt{2}}$$

Let's begin with the substitution suggested in the question. We have $$\cos t=\frac{\sqrt{1+x^4}} {1+x^2}$$ ($\theta$ is replaced by $t$ to reduce burden of typing mathjax). Differentiating we get \begin{align} -\sin t \, dt&=\dfrac {(1+x^2)\cdot \dfrac{4x^3} {2\sqrt{1+x^4}} - 2x\sqrt{1+x^4}} {(1+x^2)^2} \, dx\notag \\ &=\dfrac {(1+x^2)\cdot \dfrac{2x^3} {\sqrt{1+x^4}} - 2x\sqrt{1+x^4}} {(1+x^2)^2} \, dx\notag \\ &= 2x\cdot\dfrac {(1+x^2)\cdot \dfrac{x^2} {\sqrt{1+x^4}} - \sqrt{1+x^4}} {(1+x^2)^2} \, dx\notag \\ &= 2x\dfrac { \dfrac{x^2(1+x^2)-(1+x^4)} {\sqrt{1+x^4}}} {(1+x^2)^2} \, dx\notag \\ &= 2x\dfrac { \dfrac{x^2-1} {\sqrt{1+x^4}}} {(1+x^2)^2} \, dx\notag \\ \implies \sin t \, dt&=\frac{2x(1-x^2)}{(1+x^2)^2\sqrt{1+x^4}}\, dx\notag\\ \end{align} You need to double check your calculations as there is no cancellation in above to reduce the $(1+x^2)^2$ in denominator to $1+x^2$.

Next we note that $$\sin t =\frac{\sqrt {2}x}{1+x^2}$$ and therefore $$dt=\sqrt{2}\cdot\frac{1-x^2}{(1+x^2)\sqrt{1+x^4}}\,dx$$ or $$\frac{1-x^2}{(1+x^2)\sqrt{1+x^4}}\,dx=\frac{dt}{\sqrt{2}}$$ Integrating the above (noting that interval $[0,1]$ for $x$ maps to $[0,\pi/4]$ for $t$) we get $$I=\int_0^1\frac{1-x^2}{(1+x^2)\sqrt{1+x^4}}\,dx=\frac{1}{\sqrt {2}}\int_0^{\pi/4}\,dt=\frac{\pi}{4\sqrt{2}}$$