Groups of matrices in which all elements have all eigenvalues equal in modulus
The answer by Misha is excellent. Let me however give a more concise and self contained answer, which the OP could use when writing his paper.
First note that the statement is true for $G$ iff it is true for $\{\alpha g\mid g\in G,~\alpha\in \mathbb{R}^*\}$. Since $\text{SL}_3(\mathbb{R})\to \text{PGL}_3(\mathbb{R})$ is onto, it is enough to prove:
$(*)$ Let $G<\text{SL}_3(\mathbb{R})$ be a subgroup such that all eigenvalues of all elements are of absolute value 1. Then $G$ is conjugated to $\text{SO}(3)$.
Note that for $g\in G$ the eigenvalues are $1,\alpha,\bar{\alpha}$ for some $\alpha\in S^1\subset\mathbb{C}$. In particular $\text{tr}(g)=\text{tr}(g^{-1})$. It follows that $G$ is not Zariski dense. Its Zariski closure must be reductive by the irreducibility assumption, so it is contained in a conjugate of $\text{SO}(3)$ or $\text{SO}(2,1)$ - these are the irreducible reductive subgroups. We are left to contradict the second possibility.
Recalling that the connected component of $\text{SO}(2,1)$ is isomorphic to $\text{SL}_2(\mathbb{R})$ and by passing to a finite index subgroup of $G$, let me deal now with statement $(*_2)$, the 2-dim analogue of $(*)$.
I will explain why if $G<\text{SL}_2(\mathbb{R})$ satisfying the conditions above is Zariski dense then it is also dense wrt the usual topology, which will give a contradiction.
Assume $G$ is Zariski dense. Find $g\in G$ of infinite order. Observe that $g$ is not unipotent. Indeed, otherwise a generic conjugate of it $h\in G$ will be another unipotent and the product of high powers $g^nh^m$ will have $\text{tr}>2$ (this becomes clear when you find a basis in which $g,h$ are represented in upper,lower triangular matrix forms). It follows that $g$ is eliptic, hence, up to a choice of basis, $g\in\text{SO}(2)$ is an irrational rotation. It folows that $\text{SO}(2)<\bar{G}$. Since $\text{SO}(2)$ is a maximal subgroup, $\bar{G}=\text{SL}_2(\mathbb{R})$.
I will consider finitely generated subgroups G of $SL(3,R)$ as it suffices. Since all eigenvalues are assumed to be on the unit circle you have a dichotomy.
The subgroup G contains an elliptic element of infinite order. Then the subgroup I'd not discrete. Take its topological closure. It is a Lie subgroup of positive dimension. Consider the identity component. By inspection, it is either reducible, and so is your subgroup, or it is irreducible compact, or it contains a subgroup locally isomorphic to $SL(2,R)$. In the latter case the original group contains an element with an eigenvalue not on the unit circle. If the compact subgroup us irreducible, then G itself is relatively compact. In any case, you are done.
All eigenvalues are roots of unity. Since G is finitely generated, it contains a finite index subgroup where all elements are unipotent. Thus finite index a subgroup is contained in upper triangular group (up to conjugation). Then you are done again.
Edit. Here are some details. Below and above $R={\mathbb R}$ and $C={\mathbb C}$.
Most ingredients of the proof follow closely the proof of the Tits’ alternative as presented in our book) with Drutu, except few things are easier since you are dealing with subgroups of $SL(3,R)$.
First, some terminology: A subgroup $G$ of $SL(n,C)$ is called distal if all eigenvalues of all elements of $G$ have absolute value 1 (such matrices are also called distal). An element of $SL(n,C)$ is called quasiunipotent if all its eigenvalues are roots of unity.
Here is an elementary lemma that I will use repeatedly:
Lemma 1. If $N< SL(3,R)$ is a nontrivial connected nilpotent Lie subgroup then $N$ either preserves a unique line $L$ or a unique plane $P$ in $R^3$. In particular, if a group $G$ normalizes $N$ then $G$ also preserves that line/plane.
Below, $G< SL(3,R)$ is distal and finitely generated.
Case 1. All its elements of $G$ are quasiunipotent.
One can shorten a bit the next chain of lemmas if one is willing to use Borel’s theorem that every finitely generated subgroup $G$ of $SL(n,R)$ contains a neat subgroup $G_0$ of finite index, i.e. a subgroup such that each quasiunipotent element of $G_0$ is unipotent. Assuming this, we see that if all elements of $G$ are quasiunipotent, then $G$ contains a finite index subgroup $G_1<G$ such that all elements of $G_1$ are unipotent. Hence, by Kolchin’s theorem, $G_0$ is conjugate to the group of upper triangular matrices.
Instead of relying upon Borel and Kolchin's theorems, one can give a direct argument (the proofs are elementary, you can find them in our book for instance):
Lemma 2. If $G< GL(n,C)$ is a finitely generated subgroup such that all elements of $G$ are quasiunipotent then the orders of roots of unity appearing as eigenvalues are uniformly bounded above.
Lemma 3. If $G< GL(n,C)$ is a finitely generated subgroup such that all elements of $G$ are quasiunipotent with uniformly bounded orders of roots of unity appearing as eigenvalues, then $G$ contains a finite index subgroup $G_1$ which is conjugate to the group of upper triangular matrices. (The proof of this lemma is similar to the proof of Kolchin’s theorem.)
Corollary. If $G< GL(n,C)$ is finitely generated and each element of $G$ is quasiunipotent then $G$ is virtually nilpotent.
Now, if $G< SL(3,R)$ and all its elements are quasiunipotent, then, using Lemma 1, we see that $G$ is either finite or preserves a proper subspace of $R^3$.
Case 2. Suppose that $G< SL(3,R)$ is distal, but some $g\in G$ is not quasiunipotent. By looking at the complex Jordan normal form of such $g$ you see that it has to be diagonal (over ${\mathbb C}$), which means that $g$ is elliptic, i.e. is conjugate to a rotation $h\in SO(3)$; and this rotation has infinite order. (Here the argument is simplified by the fact that we have $3\times 3$ matrices.) Therefore, the group $G$ cannot be discrete. Consider its closure $cl(G)$ (in the classical topology) in $SL(3,R)$. Let $H$ denote the identity component of the closure (which has to be a Lie subgroup of positive dimension). The subgroup $H$ is normalized by $G$. Let $N< H$ be the nilpotent radical. Since it is a characteristic subgroup, it is also normalized by $G$. Therefore, by Lemma 1, $G$ preserves a proper subspace in $R^3$. Thus, assume that $H$ is reductive. Again, by appealing to Lemma 1 we see that $H$ has to be semisimple.
There are not that many semisimple proper connected subgroups in $SL(3,R)$, they are are $SO(2,1)$ and $SL(2,R)$ and $SO(3)$ (up to conjugation).
If $H\cong SO(3)$ then it preserves a unique inner product on $R^3$, from which it follows that $G$ also preserves the same inner product and, hence, $G$ is relatively compact in this case. Suppose that $H$ is locally isomorphic to $SL(2,R)$ or $SL(3,R)$. By density of $G$ in $H$, it follows that $G$ is not distal, a contradiction.
Lastly, here is how one can handle the case of subgroups $G$ which are not finitely generated. Represent $G$ as a union of finitely generated subgroups $G_i$, $i\in I$.
(a) If the closure of some $G_i$ contains a nontrivial semisimple subgroup $K$, this subgroup has to be compact. Then you can see that the closure of each $G_j$ containing $G_i$ also contains $K$, from which you can see that each $G_j$ preserves the unique $K$-invariant inner product. Hence, $G$ is relatively compact.
(b) Assume that $G$ is not relatively compact. Then there exists $G_i$ which preserves a unique proper subspace $V$ in $R^3$. Since each $G_j$ containing $G_i$ also preserve a unique proper subspace, $V$ has to be invariant under each $G_j$ and, thus, $V$ is $G$-invariant. qed
Last remark: There are examples of finitely generated subgroups $G$ of $SL(n,C)$ such that each element of $G$ is elliptic (conjugate to a unitary matrix) but $G$ itself is not relatively compact. This is something to be aware of.
The specific question for $SL(3,\mathbb{R})$ has already been answered by Misha and Uri Bader. I just want to make some additional remarks which say that this holds in greater generality.
Lemma: Suppose $\Gamma \subset GL(n,\mathbb{C})$ is a subgroup such that all the eigenvalues of all the elements of $G$ have absolute value $1$. Then there exists a closed subgroup $G\subset GL(n,\mathbb{C})$ containing $\Gamma $ which contains a unipotent normal subgroup $U$ such that $G/U$ is compact.
Clearly, the eigenvalue of any element of such a $G$ will have absolute value $1$, since the eigenvalue of a unipotent element is $1$ and elements of a compact group have eigenvalues of absolute value $1$. Hence the above Lemma gives an if and only if condition for $\Gamma $ to have all eigenvalues of absolute value $1$ . (An alternative way to say this is: $\Gamma $ is contained-not necessarily as a closed subgroup- in a closed amenable subgroup of $GL(n,\mathbb{C})$, which is a unipotent-by- compact extension).
Proof of the lemma: It is enough to consider the case when $\Gamma$ acts irreducibly on $\mathbb{C}^n$, since the general case follows from this by looking at a $\Gamma $ stable flag in $\mathbb{C}^n$ , where the successive quotients are irreducible. Then the sub-algebra of $M_n(\mathbb{C})$ generated by the elements $\gamma \in \Gamma$ is all of $M_n(\mathbb{C})$ by Burnside's theorem. The trace form on $M_n(\mathbb{C})$ is non-degenerate. Fix a basis of $M_n(\mathbb {C})$ of the form $g_i\in \Gamma $ and let $\epsilon _i$ be the dual basis with respect to the trace form. For every $g\in \Gamma $, we have (see Lemma (2.2) of Tits' article on Free Groups in Journal of Algebra (for a reference see the review
http://www.ams.org/mathscinet-getitem?mr=286898 ))
$$ g=\sum trace (gg_i)\epsilon _i.$$ All the elements $g\in \Gamma $ have $bounded$ entries since $trace (gg_i)$ - being a sum of eigenvalues of $gg_i$ - is bounded by the dimension $n$. This proves that $\Gamma $ is contained in a compact subgroup of $GL(n, \mathbb{C})$ if it acts irreducibly on $\mathbb{C} ^n$ and all elements have eigenvalues only of norm $1$.
Now if $\Gamma \subset GL(n, \mathbb{C})$ has the property that the ratio of any two eigenvalues has norm one, then by going to the adjoint representation of $GL(n,\mathbb {C})$ we have: $Ad(\Gamma )$ is contained in an extension of a compact group by a unipotent group. Hence the same holds for $\Gamma $ as well (up to multiplication by scalars).