Determine a constant for a certain limit involving arctan function
Because for all $x>0$ we have $0<\arctan x<x$ we conclude by a simple induction argument that the sequence $(x_n)_{n\ge0}$ is positive decreasing, so it must converge to some limit $\ell$ that satisfy $\ell=\arctan \ell$, that is $\ell=0$. Thus, $$\lim_{n\to\infty}x_n=0\tag{1}$$ It follows that $$\lim_{n\to\infty}\frac{x_{n+1}}{x_n}=\lim_{n\to\infty} \frac{\arctan{x_n}}{x_n}=1\tag{2}$$ Now, $$\eqalign{\frac{1}{x_n^2}-\frac{1}{x_{n+1}^2}&=\frac{x_n^2}{x_{n+1}^2}\cdot\frac{(\arctan x_n)^2-x_n^2}{x_n^4}\cr &=\frac{x_n^2}{x_{n+1}^2}\cdot\frac{(x_n-x_n^3/3+O(x_n^5))^2-x_n^2}{x_n^4}\cr &=\frac{x_n^2}{x_{n+1}^2}\cdot\left(-\frac{2}{3}+O(x_n^2)\right) }$$ It follows that $$\lim_{n\to\infty}\left(\frac{1}{x_{n+1}^2}-\frac{1}{x_{n}^2}\right)=\frac{2}{3}\tag{3}$$ Using Stolz-Cesàro theorem we conclude that $$\lim_{n\to\infty}\frac{1}{nx_n^2}=\frac{2}{3}\tag{4}$$ Or $$\lim_{n\to\infty}\sqrt{n}x_n=\sqrt{\frac{3}{2}}$$ Which is the desired conclusion.
There are general results for $x_{n+1}=f(x_n)$ where $f(x) = x - a_k x^k + \dots$ including special cases like iteration of sine or another iteration of sine. The key is knowing $k>1$, the index of the first power of $x$ with non-zero coefficient after $x$ itself. This is covered in the book "Asymptotic methods in analysis" by N. G. de Bruijn.