Evaluate $\iint_{[0,1]^2}\frac{dxdy}{(1+x^2+y^2)^{3/2}}$

First, note that the two integrals in the last line are equal. Just substitute $\phi=\frac{\pi}{2}-\varphi$ and see. Therefore

$$I=2\int_{0}^{\frac{\pi}{4}}\left(1-\frac{\cos\phi}{\sqrt{2-\sin^{2}\phi}}\right){\rm d\phi}=\left[\begin{matrix}\sin\phi=\sqrt{2}\sin\theta\\\cos\phi{\rm d}\phi=\sqrt{2}\cos\theta{\rm d}\theta\end{matrix}\right]=$$

$$=\frac{\pi}{2}-2\int_{0}^{\frac{\pi}{6}}\frac{\cos\theta}{\sqrt{1-\sin^{2}\theta}}{\rm d}\theta=\frac{\pi}{2}-\frac{\pi}{3}=\frac{\pi}{6}$$

Here is a solution that takes the issue right at its beginning. Your idea to turn to polar coordinates and divide the integration domain into two triangles $OAB$ and $OBC$ (with $A(1,0),B(1,1)$ and $C(0,1)$) is good. You should do it at once. One gets :

$$I=\iint_{[0,1]^2}\frac{dxdy}{(1+x^2+y^2)^{3/2}}=I_1+I_2$$

where

$$I_1=\int_{\theta=0}^{\theta=\pi/4}\int_{r=0}^{r=1/\cos{\theta}}\frac{1}{(1+r^2)^{3/2}}\color{red}{r}drd\theta=$$

and $I_2$ is in fact equal to $I_1$ for an evident symmetry reason. (please note that $\color{red}{r}$ is the jacobian of this change of variables). Thus :

$$I=2I_1=2\int_{\theta=0}^{\theta=\pi/4}[-\frac{1}{(1+r^2)^{1/2}}]_{r=0}^{r=1/\cos(\theta)} d\theta$$

giving :

$$I=2\int_{\theta=0}^{\theta=\pi/4}(1-\frac{1}{(1+(1/\cos(\theta)^2)^{1/2}})d\theta=2\left(\frac{\pi}{4}-\int_{\theta=0}^{\theta=\pi/4}\frac{\cos(\theta)}{\sqrt{1+\cos(\theta)^2}}d\theta\right)$$

where the remaining integral, transformed into :

$$\int_{\theta=0}^{\theta=\pi/4}\frac{\cos(\theta)}{\sqrt{2 -\sin(\theta)^2}}d\theta$$

is readily integrated using antiderivative :

$$\sin^{-1}\left(\tfrac{1}{\sqrt{2}}\sin(\theta)\right)$$

giving finally :

$$I=2(\frac{\pi}{4}-\frac{\pi}{6})=\frac{\pi}{6}$$