Converting double integrals to polar form: what would the limits be here?

Note that in order to cover $\mathbb{R}^2$, $r$ extends from $0$ to $\infty$ and $\theta$ spans an entire period of $\sin(\theta)$ and $\cos(\theta)$. (For example, the first quadrant alone is covered by $\theta \in [0,\pi/2]$, $r\in [0,\infty)$.)

Hence,

$$\begin{align} \int_{-\infty}^\infty\int_{-\infty}^\infty\frac{x^2}{(1+\sqrt{x^2+y^2})^5}\,dx\,dy&=\int_0^{2\pi}\int_0^\infty \frac{r^2\cos^2(\theta)}{(1+r)^5}r\,dr\,d\theta\\\\ &=\underbrace{\left(\int_0^{2\pi}\cos^2(\theta)\,d\theta\right)}_{=\pi}\underbrace{\left(\int_0^\infty \frac{r^3}{(1+r)^5}\,dr\right)}_{=1/4}\\\\ &=\frac{\pi}{4} \end{align}$$

By symmetry (we are free to exchange $x$ and $y$) the given integral equals: $$ \frac{1}{2}\iint_{\mathbb{R}^2}\frac{x^2+y^2}{(1+\sqrt{x^2+y^2})^5}\,dx\,dy = \pi \int_{0}^{+\infty}\frac{r^3}{(1+r)^5}\,dr = \color{red}{\frac{\pi}{4}}.$$ Your $\theta$ variable simply ranges over $(0,2\pi)$.