If $G$ is not abelian, then $\#\text{Inn}(G) \geq 4$

The contrapositive is much clearer:

If $\#\text{Inn}(G) < 4$, then $G$ is abelian

The key facts are

• $\text{Inn}(G) \cong G/Z(G)$

• All groups of order less than $4$ are cyclic

• If $G/Z(G)$ is cyclic then $G$ is abelian

As you identified in the comments, the key to this is the isomorphism

$$G/Z(G)\cong{\rm Inn}(G).$$

A proof of this isomorphism can be found in Gallian's "Contemporary Abstract Algebra (Eighth Edition)", page 194, Theorem 9.4.