An algebraic number is not a root of unity?

Assume that the roots $\zeta$ and $\xi^2/\zeta$ of your equation are roots of unity. Then by Vieta Theorem $\zeta+\xi^2/\zeta=-(\xi^2-\xi+1)$; equivalently sum of 5 roots of unity $\zeta+\xi^2/\zeta+\xi^2-\xi+1$ vanishes. All such relations are classified in Theorem 6 of paper by Conway and Jones. This implies the result.


You are interested in the solutions of the bivariate equation $V\colon f(X,Y)=X^2+X(Y^2-X+1)+Y^2=0$ in roots of unity, a sort of question known as "Manin-Mumford problem". The answer is known in the greater generality of subvarieties of algebraic groups — except for finitely many points, they are explained by the existence of "torsion subvarieties".

In the case you're interested in, the answer has been proved by Ihara, Serre, Tate, see Lang, Division points on curves. The proof is quite elementary. Let $z=(\xi,\eta)$ be a point of $V$ where $\xi,\eta$ are roots of unity; let $n$ be the lcm of their orders. Then, for every integer $d\geq2$ which is prime to $n$, there exists $\sigma\in\mathop{\rm Gal}(\overline{\mathbf Q}/\mathbf Q)$ such that $\sigma(z)=z^d$. Then, $z$ is a common root of $f(X,Y)$ and of $f(X^d,Y^d)$. By Bézout's theorem, the locus defined by these two equations is either equal to $V$ (if $f(X,Y)$ divides $f(X^d,Y^d)$), or is finite, of cardinality $\leq \deg(f)^2 d=9d$. In the first case, $f$ (which is irreducible) would be the difference of two monomials, which it isn't. So we are in the latter case. Since all conjugates of $z$ belong to this finite set, one gets the inequality $\phi(n)\leq 9d$.

By the prime number theorem, it is possible to choose $d$ quite small with respect to $n$, say $d\ll \log(n)$. (The worst case is when $n=p_1\dots p_r$, the product of the $r$ first primes—then, $d=p_{r+1}$ is possible, and (PNT), one has $p_{r+1}\sim r\log(r)$ while $\log(n)=\sum_{i\leq r} \log(p_i)\sim r\log(r)$ as well, so that $p_{r+1} \approx \log(n)$.) Similarly, $\phi(n)\gg n^{1-\epsilon}$, hence $n^{1-\epsilon}\ll 9 \log(n)$. This bounds $n$ from above, and it remains to check every possible $z$ in a finite set.


I assume that $n=p$ be a prime and $p>3$: indeed, if $p=3$ then your statement is false because $\xi^2-\xi+1=0$. Your polynomial is $$ \chi(\lambda)=\lambda^2+(\xi-1)^2\lambda+\lambda\xi+\xi^2\in\mathbb{Z}[\xi] $$ where $\xi$ is a primitive $2p$-th root of unity. Let $\zeta$ be a root of $\chi$ and assume it is a root of unity: hence, as Sebastian Schoennenbeck observed, either $\zeta\in\mathbb{Q}(\xi)=\mathbb{Q}(\mu_{2p})$ or it lies in the quadratic extension $\mathbb{Q}(\mu_{4p})$. The latter is not the case since the minimal polynomial of $\zeta$ over $\mathbb{Q}(\mu_{2p})$ would be $\lambda^2-\xi$, therefore $\zeta\in \langle\xi\rangle$, the cyclic group of order $2p$ generated by $\xi$ inside $\mathbb{Z}[\xi]^\times$. To show that this is not the case, observe that all roots of unity in $\mathbb{Z}[\xi]$ reduce to $\pm 1$ modulo the maximal ideal $\mathfrak{p}\subseteq \mathbb{Z}[\xi]$ which lies above $p$ and contains (is in fact generated by) $\xi-1$ (see Proposition 2.8 of Washingtons's Introduction to Cyclotomic Fields). If we reduce the polynomial$\mod\mathfrak{p}$ we thus find $$ \bar{\chi}(\lambda)=\lambda^2\pm \lambda+1 $$ and neither $1$ nor $-1$ are roots of either of these polynomials unless $p=3$ which we excluded. Hence, no roots of unity can be a root of $\chi$.

The argument above extends immediately to the case when $n=p^e$ is a prime power but does not extend to the case where $n=p^aq^b$ is composite because $\xi-1$ becomes a unit.