What is this notation? Cyclic group $\mathbb{Z}^*_8$

$\mathbb{Z_8}^*$ denotes the multiplicative group of $\mathbb{Z_8}$ as Mark Bennet has said.

You can show that $x \in \mathbb{Z_n}$ has a multiplicative inverse if and only if $(x,n)=1$. The proof is based on a special case of Bezout's theorem that states $(x,n)=1$ if and only if $\exists a,b \in \mathbb{Z}: ax+bn = 1$.

If $(x,n)=1$, then $\exists a,b: ax+bn=1$. This implies that $bn = 1-ax$ or $n \mid 1-ax$ which is the same as $ax \equiv 1 \pmod{n}$.

On the other hand, if there exists $a \in \mathbb{Z_n}$ such that $ax \equiv 1 \pmod{n}$ then $\exists b \in \mathbb{Z}: bn = 1 - ax$. Which gives you the converse.

So, the necessary and sufficient condition for an element in $\mathbb{Z_n}$ to be invertible is that it is relatively prime to $n$.

Taking the elements coprime with n means those who have no common prime factor with n. This means we take only those who have an inverse (and are not zero divisors, which is saying the same thing dealing with finite groups). In the first example with $\mathbb{Z}/8$ , we keep the classes representatives as 1, 3, 5, 7 and make them into a abelian multiplicative group named $\mathbb{Z}^*_8$ . Some use an *, others like Andrew Baker from University of Glasgow use a x. He would write $(\mathbb{Z}/8)^×$. There is a group isomorphism with $\mathbb{Z}^*_8$ and $\mathbb{Z}/2$ x $\mathbb{Z}/2$.

This shows the isomorphism:

http://www.maths.gla.ac.uk/~ajb/dvi-ps/padicnotes.pdf