formula for square of absolute value of difference of two variables $|a-b|^{2}$

We have $|a-b|^2=(a-b)^2$. No difference! For $|a-b|=a-b$ if $a\ge b$, and $|a-b|=-(a-b)$ if $a\lt b$. Now recall that in general $(-x)^2=x^2$.

Remark: In our answer, we assumed that $a$ and $b$ are real numbers, or variables that range over the reals. In the complex numbers, there is a notion of absolute value, usually called the norm of the complex number. In that setting, the answer becomes more complicated.

Yes, $$ |a-b|^2=(a-b)^2. $$ The left hand side is just the number $x=|a-b|$ squared, while the right hand side is the number $y=(a-b)$ squared. But $x$ and $y$ differ at most by a sign, i.e. $x=y$ or $x=-y$ and hence if you square them, they are equal.