How to prove that the set is not a regular surface?

Consider the upper cone $S_1$ where $$x^2 + y^2 = z^2$$ with $z\geq0$ and the lower cone $S_2$ where $$x^2 + y^2 = z^2$$ with $z\lt0$. Notice that any open set in $\mathbb{R^3}$ containing the vertex $(0,0,0)$ must also contain points of $S_1$(besides the vertex) and of $S_2$. We have that $S = S_1 \bigcup S_2$. Suppose there is an open set $U$ in $\mathbb{R^2}$, an open set $W$ in $\mathbb{R^3}$ containing $(0,0,0)$, and an homeomorphism $H:U\to S\cap W$. If $a,b$ and $c$ are three distinct points in $U$ such that $H(a)=u$, $H(b)=(0,0,0)$ and $H(c)=v$, with $u\in S_1$ and $v \in S_2$. You can find a path connecting $a$ and $c$ in $U$ such that $b$ is not in this path, but as $H$ is an homeomorphism, it means that the image of this path by $H$ does not pass through the vertex, i.e., the image of this path by $H$ would connect a point in $S_1$ (different from the vertex) and a point of $S_2$ but not contain the vertex. Contradiction.

Note that if S where to be a regular surface, then it would be, in an open nhood of $(0,0,0)$ in $S$, the graph of differentiable function of the form: $x=f(y,z)$, $y=g(x,z)$ or $z=h(x,y)$. But that obviously cannot be the case, once the projections of S onto the coordinate planes isn't injective.

Your surface is given by an equation, in this case $G(x,y,z) = x^2+y^2-z^2$, $S = \{(x,y,z)\in\mathbb{R}^3|G(x,y,z) = 0\}$. A surface is regular if for every point $p\in S$ you have that $DG|_p$ is regular (i.e. it has full rank).

You also have that a $C^k$ surface is locally the graph of a $C^k$ function above one of the coordinate planes.

If you want to go deeper on the subject, here's an introduction to submanifolds of Euclidean space (surfaces in $\mathbb{R}^3$ are a special case): Submanifolds of Euclidean Space