Check whether $G$ is group or not

The operation $*$ is not associative, since:\begin{align}\bigl||2-1|-1\bigr|&=0\\&\neq2\\&=\bigl|2-|1-1|\bigr|\end{align}and therefore, $(G,*)$ is not a group.

The Cayley table (that also shows closure, identity and inverses) is \begin{array}{c|ccc} * & 0 & 1 & 2 \\ \hline 0 & 0 & 1 & 2 \\ 1 & 1 & 0 & 1\\ 2 & 2 & 1 & 0 \end{array} Since in the second row the element $1$ appears twice, the set is not a group.

In a group, from $xy=xz$ one deduces $y=z$; here, instead, $1*0=1*2$, but $0\ne2$.