Groups without factorization

This must be "well-known": If we have $G = AB$ when $G$ is a finite group, and $A,B$ are proper subgroups of $G$, then we may suppose that $A$ and $B$ are both maximal.

For if $A$ is not maximal, and $A < C$ with $C$ maximal, then we still have $G = CB,$ and $B \not \leq C$, so we may replace $A$ by $C$ and assume that $A$ is maximal.

Now if $B$ is not maximal, but $B < D$ with $D$ maximal, then we have $G = AD$ and $A,D$ are both proper and maximal, so we can replace $B$ by $D$ and assume, $A,B$ are both maximal.

Hence the finite group $G$ has no factorization if and only if $|AB| < |G|$ whenever $A$ and $B$ are maximal subgroups of $G$, ie if and only if $[G:A] > [B:A \cap B]$ whenever $A,B$ are maximal subgroups of $G$.

In particular, since the Frattini subgroup $\Phi(G)$ is the intersection of the maximal subgroups of $G$ we conclude that $G$ has a factorization if and only if $G/\Phi(G)$ has a factorization.

Since $G/\Phi(G)$ has trivial Frattini subgroup, we conclude from the argument you gave in the question, that if $G$ has no factorization, then $G/\Phi(G)$ is a simple group.

Later edit: By the way, this answers the subquestion in the negative. If we take a nonsplit extension $H$ of a non-Abelian finite simple group $G$ which itself has no factorization by a finite irreducible $G$-module $M$, then we obtain a non-simple finite group $H$ which is neither simple nor cyclic of prime-power order, yet there is is no factorization of $H$ (and there are examples of such non-split extensions- note that $\Phi(H) = M$ in that case. For example, $J_{3}$ admits no factorization, and it has a (non-split) triple cover which also admits no factorization).

Even later edit: In fact, we can now further restrict all finite groups $G$ which admit no factorization. By a slight extension of the argument above, we deduce that $G$ has no factorization if and only if $G/X$ has no factorization whenever $X \lhd G$ with $X \leq \Phi(G)$.

If G has no factorization, we know that $G/\Phi(G)$ is a simple group without a factorization.

If $G/\Phi(G)$ is cyclic of prime order, then $G$ is cyclic, and $G$ must be a cyclic $p$-group for some prime $p$.

If $G/\Phi(G)$ is non-Abelian simple, then we may take $X \lhd G$ with $X < \Phi(G)$ and $[\Phi(G):X]$ minimal. Let $V = \Phi(G)/X.$

Then $V = \Phi(G/X)$ and $G/X$ is a non-split extension of the simple group $G/\Phi(G)$ by the irreducible $G/\Phi(G)$-module $V.$

Hence the finite group $G$ admits no factorization if and only if either $G$ is cyclic of prime power order or else $G/\Phi(G)$ is a non-Abelian simple group with no factorization which admits a non-split extension by a finite (possibly trivial) irreducible $G/\Phi(G)$-module $V$ occurring as a "top" $G$-chief factor within $\Phi(G)$ (that is to say $V$ occurs as $\Phi(G)/X$ where $X$ is maximal subject to being normal in $G$ and properly contained in $\Phi(G)$).

The smallest non-abelian group without factorization is simple of order $1092$: it is $A_1(13)$.

Using the answer of Geoff and by browsing the book The maximal factorizations of the finite simple groups and their automorphism groups (by Martin W. Liebeck, Cheryl E. Praeger and Jan Saxl), the finite simple groups without factorization are the following:

• Every cyclic group of prime order.
• Alternating: none
• Chevalley groups: $A_n(q)$ with $n=1$, $q \equiv 1 (\textrm{mod}\ 4)$ and $q\neq 5,9,29$, or with $n \ge 2$ even and $(n+1) | (q-1)$; $E_6(q)$; $E_7(q)$; $E_8(q)$; $F_4(q)$ with $q \neq 2^n$; $G_2(q) $ with $q>4 $ and $q \neq 3^n$.
• Twisted Chevalley groups: $^2A_{2n}(q^2)$ except $(n,q) = (1,3),(1,5),(4,2)$; $^2B_2(2^{2n+1})$; $^2D_{2n}(q^2)$; $^3D_4(q^3)$; $^2E_6(q^2)$; $^2F_4(2^{2n+1})$; $^2G_2(3^{2n+1})$.
• Sporadic: $J_1$, $M_{22}$, $J_3$, $McL$, $O'N$, $Co_3$, $Co_2$, $F_5$, $Ly$, $F_3$, $Fi_{23}$, $J_4$, $F_{3+}$, $F_2$, $F_1$.

Before any use: this list requires a double checking.