How to prove this conjecture $x_{n}(n\ge 4)$ can't be an integer.
This answer uses that for $n\ge 4$, $$\frac{1+\sqrt{4n-3}}{2}\lt x_n\lt \frac{1+\sqrt{4n+1}}{2}\tag1$$ The proof for $(1)$ is written at the end of this answer.
Multiplying $(1)$ by $2$, subtracting $1$ and squaring, we get $$4n-3\lt 4(x_n^2-x_n)+1\lt 4n+1$$ Then, subtracting $1$ and dividing by $4$, we get $$n-1\lt x_n^2-x_n\lt n$$ Since $x_n^2-x_n$ is not an integer, we have that $x_n$ is not an integer.
So, your conjecture is true.
Let us prove $(1)$ by induction.
For $n=4$, since $13\lt 16\lt 17$, we get $$\frac{1+\sqrt{13}}{2}\lt \frac{1+\sqrt{16}}2=x_4\lt \frac{1+\sqrt{17}}{2}$$
Supposing that $(1)$ holds for some $n\ge 4$ gives $$\begin{align}x_{n+1}-\frac{1+\sqrt{4(n+1)-3}}{2}&=1+\frac{n}{x_n}-\frac{1+\sqrt{4n+1}}{2}\\\\&\gt 1+\frac{2n}{1+\sqrt{4n+1}}-\frac{1+\sqrt{4n+1}}{2}=0\end{align}$$ and $$\begin{align}\frac{1+\sqrt{4(n+1)+1}}{2}-x_{n+1}&=\frac{1+\sqrt{4n+5}}{2}-1-\frac{n}{x_n}\\\\&\gt \frac{1+\sqrt{4n+5}}{2}-1-\frac{2n}{1+\sqrt{4n-3}}\\\\&=\frac{1+\sqrt{4n+5}}{2}-1-\frac{2n(1-\sqrt{4n-3})}{1-(4n-3)}\\\\&=\frac{(n-1)\sqrt{4n+5}+1-n\sqrt{4n-3}}{2(n-1)}\\\\&=\frac{\sqrt{4n+5}-3}{(n-1)\sqrt{4n+5}+1+n\sqrt{4n-3}}\\\\&\gt 0\qquad\quad\blacksquare\end{align}$$
The conjecture is true.
Define, as in @S.C.B.'s comment, the sequence of Telephone numbers: $$ T_0 = T_1 = 1, \quad T_{n+1} = T_{n} + nT_{n-1},\ n\ge 1, \tag{1} $$ so that $$ x_n = \frac{T_n}{T_{n-1}},\ n\ge 1. $$ Then $$ T_n = \sum_{k=0}^{\lfloor n/2\rfloor} {n\choose 2k}(2k-1)!!. $$ I will prove first @S.C.B.'s claim:
Claim For each $n\ge 1$, $T_n$ and $T_{n+1}$ have no common odd prime divisor.
It relies on
Lemma For each $n\ge 1$, $T_n$ and $n$ have no common odd prime divisor.
Proof Let $p>2$ be an odd prime dividing $n$. Write $$ T_n = 1 + \sum_{k=1}^{\lfloor n/2\rfloor} {n\choose 2k}(2k-1)!!. $$ Note that $p\mid {n\choose 2k}$ for $1\le k< p/2$, and $p\mid (2k-1)!!$ for $k> p/2$. Therefore, $T_n = 1\pmod p$, yielding the statement.
Proof of the claim Let $p>2$ be a prime. Assume, by way of contradiction, that $n$ is the smallest positive integer such that $p$ divides both $T_{n+1}$ and $T_n$. Then it follows from $(1)$ that $p\mid n$, contradicting Lemma.
Therefore, the only possibility for $x_n$ to be an integer is $T_{n-1}$ being a power of $2$. However, as it is explained in the Wikipedia page, this is not possible for $n\ge 4$.