Making sense of the commutator
We can't really say "how non-commutative" $a$ and $b$ are, without some corresponding notion of "how much not the identity" any given element of a group $G$ is, as you point out. For some groups, we may be able to do this, but in general, there's no "universal" way.
The real value in this, is not the individual commutators $[a,b]$, but rather the commutator subgroup $[G,G]$. It should be clear $G/[G,G]$ is abelian, for:
$(x[G,G])(y[G,G])(x[G,G])^{-1}(y[G,G])^{-1} = [x,y][G,G] = e[G,G] = [G,G]$
But the story doesn't end there, if $N$ is any normal subgroup such that $G/N$ is abelian, we have $[G,G] \subseteq N$. The reason is very plain:
if for any $x,y \in G$, we have $(xN)(yN) = xyN = yxN = (yN)(xN)$, then we must have $xy(yx)^{-1} = [x,y] \in N$ for any pair $x,y \in G$. Thus $[G,G]$ is minimal among all normal subgroup $N$ that make $G/N$ abelian.
Another nice thing about this, is that the way we do it doesn't really depend on the group $G$ in the following sense: if $\phi:G \to H$ is a group homomorphism, we get a group homomorphism $\tilde{\phi}:G/[G,G] \to H/[H,H]$ of abelian groups defined by:
$\tilde{\phi}(x[G,G]) = \phi(x)[H,H]$, since a homomorphism preserves commutators:
$\phi([x,y]) = [\phi(x),\phi(y)]$.