Show that the group is not simple

Suppose $G$ is simple and let $1<m\leq 4$ be the index of a subgroup $H$. Then the action of $G$ on the set of left cosets of $H$ yields a nontrivial homomorphism $G\to S_m$, and since $G$ is simple the homomorphism is injective, so $|G|\le 24$. The only simple groups with order this small are of prime order.

• There is a mistake in your argument (from EDIT), you assumed $|G| | m! $ then $G$ is simple. Here, to get your argument fixed, you should first suppose by contradiction (original question) that $G$ is simple, then use the counterpositive of your Proposition. That is, knowing that $H \leq G$ and $[G:H] = m$ and assuming $G$ simple, you have $|G| | m!$. And work on each case as you did so far to get a contradiction.

• Yes, $\mathbb Z_6$, is abelian so every subgroup is normal in $\mathbb Z_6$ and $S_3$, are not simple because of what you have pointed out.

• Case $|G| = 8$ we have that $G$ is not simple because it is a $2$-group and has centre $Z(G)$ non-trivial, which is normal in $G$.

• Case $|G| = 12 = 2^2 \cdot 3 $. Looking at the number Sylow $3$-subgroups of $G$ we have that $n_3 = 1$ or $4$ (). If it is 1, then it is normal ($2^{nd}$ - Sylow Theorem) so $G$ is not simple. We assume then $n_3 = 4$. Then as each of those subgroups has order $3$, which is prime, thus they must be disjoint, therefore there $8 \cdot 2 = 8$ elements of order $3$. Which leaves us with $4$ elements left, and they must be all the elements of the Sylow $2$-subgroup (again $2^{nd}$ - Sylow Theorem). We conclude that latter must be unique and therefore normal.

• Case $|G| = 24 = 2^3\cdot 3$ similarly we have that $$ \begin{cases}n_3 \equiv 1 \mod 3 , n_3 | 8 \\ n_8 \equiv 1\mod 8 , n_8 | 3\end{cases} \implies \begin{cases}n_3 = 1 \,\,\ \text{or} \,\,\ n _3 =4 \\ n_8 = 1\,\,\, \text{or} \,\,\, n_8 = 3\end{cases}$$

Now if either $n_3 = 1$ or $n_8 =1 $ we are done. Now if $n_8 = 3$ then the action of G by conjugation on its subgroups of order $8$ determines $\varphi : G \to S_3$. By the $2^{nd}$-Sylow Theorem the image of $G$ acts in $S_3$ transitively on the set $3$ subgroups of order $8$. Then the image is not trivial and $\ker \varphi \neq G$. As $|S_3|= 6 < |G| = 24 $ we have that $\ker \phi \neq \{e\}$. We may conclude that $G$ is not simple.