Every group of order $150$ has a normal subgroup of order $25$

As you say, by Sylow's theorem, either $n_5=1$ or $n_5=6$. If $n_5=1$, we have found a normal subgroup of $25$, so we are done. So let's assume that $n_5=6$. We will exhibit a normal subgroup of order $5$ as follows. Let $P$ and $Q$ be distinct Sylow $5$-subgroups. Using the fact that $|PQ| = \frac{|P|\cdot |Q|}{|P\cap Q|}$, it is easy to see that $|P\cap Q| = 1$ is impossible! So we conclude that the subgroup $T=P\cap Q$ has order $5$.

We claim that $T$ is normal in $G$. It is a fact if $H$ is a proper subgroup of a $p$-group $L$, then $H$ is properly contained in its normalizer $N_{L}(H)$ (see here for proof). Since $P$ and $Q$ are $p$-groups, we can apply this result. Hence, $T\subsetneq N_{P}(T)$ and $T\subsetneq N_{Q}( T)$. So $N_{P}(T) = P$ and $N_{Q}(T)=Q$. This immediately gives $P\subseteq N_{G}(T)$ and $Q\subseteq N_{G}(T)$, which implies $PQ \subseteq N_{G}(T)$. Again using the formula above, $PQ=\{p \cdot q: p\in P, q\in Q\}$ (as a subset) has cardinality $125$. By order considerations, we get that $N_{G}(T)=150$, so $N_{G}(T)=G$ and $T$ is a normal subgroup of order $5$.

This proves the hint. Now let's finally show that $G$ has a normal subgroup of order $25$. So let $T$ be the normal subgroup of order $5$ constructed above. Then the quotient group $G/T$ has order $150/5=30$. Now every group of order $30$ contains a normal Sylow $5$-subgroup (see this MSE thread). Let $H$ be a normal subgroup of order $5$ in $G/T$, and consider the canonical map $\pi: G\to G/T$. Then $\pi^{-1}(H)$ is normal subgroup of $G$. By correspondence theorem, $[G: \pi^{-1}(H)]=[G/T: H]=6$, so $\pi^{-1}(H)$ is the desired normal subgroup of index $6$, i.e. desired subgroup of order $25$.

(This shows that the case $n_5=6$ is actually impossible, and there is only one subgroup of order $25$, which is the $5$-Sylow subgroup)


There are easier solutions, especially if your primary goal is to show the group is not simple. (See the paragraph starting (C) for a really simple argument.)

(A) For non-simplicity: You already know by Sylow that if the group is simple it must have $6$ $5$-Sylow subgroups. If this is the case, then it acts by conjugation on these $6$ subgroups both transitively (by Sylow) and faithfully (since it's simple), and so embeds monomorphically as a subgroup of $S_6$. As $150$ does not divide $720$ (the order of $S_6$), this is impossible. So no group of order $150$ is simple.

(B) Furthermore, doing just a little more work, we can also use this homomorphism to $S_6$ to show the $5$-Sylow of any group of order $150$ is normal. The image of the homomorphism must not be divisible by $25$ (since $25$ does not divide $720$) and the image must be divisible by $6$ (to be transitive on $6$ elements). Thus, the image has order $6$ or $30$. In either event, you can see by elementary means (and probably already know) that the image has a normal $5$-Sylow, and, since the kernel is a $5$-group, the pre-image of that in $G$ is a normal $5$-Sylow in $G$.

(The group of order $30$ has a normal $5$-Sylow for example, because it has a normal subgroup of index $2$--since any element of order $2$ acts by odd permutations by translation--$15$ $2$-cycles--, so just take the elements that act by even permutations--and this normal subgroup of order $15$ both contains all $5$-Sylows and can have only one $5$-Sylow.)

Alternatively, you can do without the last paragraph-and-a-half by showing (pretty easy) that $S_6$ has no subgroup of order $30$ so the image must have order $6$.

(C) But the simplest proof of all goes like this: $G$ must have an index $2$ subgroup, since in its action on itself by multiplication, an order $2$ element maps to an odd permutation (a product of $75$ transpositions), so the elements that map to even permutations are an index $2$ subgroup $H$, which is normal and has order $75$. By Sylow's theorem, $H$ has a unique $5$-Sylow which is thus characteristic in $H$ and hence normal in $G$.

(D) Of course, if you know Philip Hall's theorem, and you know that not simple for order $150$ implies solvable, then (A) is enough since you have immediately that the $5$-Sylow is normal in any solvable group of order $150$ since the number of $5$-Sylows in a solvable group is a product of prime powers each individually congruent to $1$ mod $5$.

This embedding in $S_n$ idea works for a bunch of similar problems as well. In more complex cases, one can sometimes make more sophisticated use of this embedding in a symmetric group to make inferences about the structure of the given group; for example, if in a simple group the number of $p$-Sylows of order $p^2$ is fewer than $p^2$, then the embedding shows they must be elementary abelian rather than cyclic. This is actually kind of a big deal--if a simple group has an element of order $p^n$ all of its subgroups must have index at least $p^n$, which is an enormous constraint on the structure of the group, especially if $p$ is large. This is why so often Sylow subgroups of simple groups are elementary abelian or extra-special.


Just to state the simplest answer alone so people see it: Since $|G|=2m$ with $m$ odd, $G$ has an index $2$ subgroup $H$ of order $75$. By Sylow, $H$ has a unique 5-Sylow $P$ of order $25$. Since $P$ is characteristic in $H$, which is normal in $G$ (because index $2$), $P$ is normal in $G$, so it is the unique $5$-Sylow of $G$.