How can I prove that every group of $N = 255$ elements is commutative?

The key to $N=255$ is the observation that it is squarefree and coprime with its totient function.

Let $n_q$ denote the number of Sylow $q$-subgroups.

So, let us begin by the observation than any group $G$ of order $15$ is cyclic. Indeed, let us merely note $n_3\mid 5$, $n_3\equiv 1\text{ mod }3$ and so $n_3=1$. Similarly, $n_5\mid 3$ and $n_5\equiv 1\text{ mod }5$ which gives that $n_3=1$. Thus, we see that both the Sylow subgroups $P,Q$ of $G$ are both normal. Since evidently $P\cap Q=\{e\}$, and $PQ=G$ (since $\displaystyle |PQ|=\frac{|P||Q|}{|P\cap Q|}=\frac{15}{1}=15=|G|$) we may conclude that $G\cong P\times Q\cong\mathbb{Z}/3\times\mathbb{Z}/5\cong\mathbb{Z}/15$.

Now, to the case of $|G|=255$. Let $P$ be a Sylow $17$-subgroup. Note then that $n_{17}\mid 15$ and $n_{17}\cong 1\text{ mod }17$ gives $n_{17}=1$. Thus, we see that $P\unlhd G$. Now, here comes the step that is different than the standard approach. Since $P$ is normal we have that $G$ acts on $P$ by conjugation, giving us a group map $G\to\text{Aut}(P)$ whose kernel is $C_G(P)$. Now, note that since $P$ is just $C_{17}$ we have that $|\text{Aut}(P)|=16$. Note now that since $(|G|,16)=1$ you can conclude from the first isomorphism theorem that the image of the group map $G\to\text{Aut}(P)$ is trivial, and so the kernel is everything. This says that $C_G(P)=G$, and so $P\subseteq Z(G)$. Note then that $G/P$ is a group of order $15$ which, by what we have said in the previous paragraph, is cyclic. Now, it is a common theorem that if you mod out a group by a subgroup of its center and you get something cyclic, your group must have been abelian the whole time. So, from all of this we can conclude that $G$ is abelian in which case, by any method that tickles your fancy, you get that $G$ must just be $\mathbb{Z}/255$.

While the above may be ad hoc, it was actually just a simple application of the following (EXTREMELY USEFUL!) theorems:

Theorem: Let $G$ be a finite group and $N\unlhd G$. Then, if $(|G|,|\text{Aut}(N)|)=1$ then $N\leqslant Z(G)$.

and

Theorem: If $G$ is a group and $N\leqslant Z(G)$ such that $G/N$ is cyclic of finite order, then $G$ is abelian.

These allow us to prove the more general theorem:

Theorem: Let $G$ be a finite group such that $(|G|,\varphi(|G|))=1$ (where $\varphi$ is the totient function), then $G$ is cyclic.

As a side note, a cool fact is that the integer $n$ has the property that the ONLY subgroup of order $n$ is $\mathbb{Z}/n$ is equivalent to $(n,\varphi(n))=1$.

The proof of the above can be found on my blog here.

Suppose $G$ is a finite group of order $|G| = 255 = 3 \cdot 5 \cdot 17$. We will show that $G$ must be cyclic.

As you noticed, there must be only one Sylow subgroup $P$ of order $17$. Thus $P$ must be a normal subgroup. Since $17$ is prime the subgroup $P$ is cyclic, say $P = \langle a \rangle$. It is not difficult to see that every group of order $15$ is cyclic, so $G/P = \langle bP \rangle$ is also cyclic. Now $(bP)^{|b|} = P$, so $|bP| = 15 \mid |b|$.

If $|b| = 255$ we are done. Therefore we can assume $|b| = 15$. We will show that $ab = ba$ which implies that $ab$ has order $255$ since $15$ and $17$ are coprime. Because $P$ is normal, $bab^{-1} = a^i$ for some integer $i$. Using this fact we get $b^2ab^{-2} = ba^ib^{-1} = (bab^{-1})^i = a^{i^2}$ and by induction $b^kab^{-k} = a^{i^k}$ for all $k \geq 1$. Thus $a = b^{15}ab^{-15} = a^{i^{15}}$, which gives $i^{15} \equiv 1 \mod{17}$. By Fermat's little theorem $i^{16} \equiv 1 \mod{17}$ and since $15$ and $16$ are coprime, $i \equiv 1 \mod{17}$. Therefore $bab^{-1} = a$, proving the claim.

This exercise is a special case of a more general fact. Using the idea of this solution and the fact that in a group of squarefree order the Sylow subgroup corresponding to the largest prime is normal, you can prove that every group of order $n$ is cyclic when $n$ and $\varphi(n)$ are coprime ($\varphi$ is the totient function). This is done by Tibor Szele in the following short article (in German).

T. Szele, Uber die endlichen ordnungszahlen zu denen nur eine Gruppe gehirt, Commentarii Mathematici Helvetici, 20, 265-67, (1947).

$$|N|=255=3\cdot 5\cdot 17$$

By Sylow theorem, the $\,17-$Sylow sbgp. $\,P_{17}\,$ is normal in $\,N\,$, so taking any $\,5-$Sylow sbgp. $\,P_5\,$, we get a sbgp. $\,K:=P_5P_{17}\,$ of index $\,3\,$ in $\,N\,$, and since this last is the minimal prime dividing $\,|N|\,$ , we get that $\,K\triangleleft N\,$ .

It's easy to prove that any group of order $\,85\,$ is cyclic , and thus its automorphism group has order $\,\phi(5)\phi(17)=64\,$ , so the only homomorphism $\,C_3\to Aut(K)\,$ ( the first group is the cyclic one of order $\,3\,$) there is is the trivial one (why?), and since by the above it follows that $\,N\cong K\rtimes P_3\,$ (with, of course, $\,P_3\cong C_3\,$), we get this semidirect product is in fact a direct one of abelian groups and, thus, $\,N\,$ is abelian and, in fact, cyclic.