Show that $a^2 \not\equiv b^2 \pmod p$

If $p\mid a^2-b^2=(a-b)(a+b)$, then $p\mid a+b$ or $p\mid a-b$ because it is prime. Now as you correctly noted $a+b\lt p$, so $p$ cannot divide $a+b$, therefore $p\mid a-b$; in other words $a\equiv b\pmod{p}$.


If

$a^2 \equiv b^2 \mod p \tag 1$

then

$a^2 - b^2 \equiv 0 \mod p, \tag 2$

or

$(a + b)(a - b) \equiv a^2 - b^2 \equiv 0 \mod p; \tag 3$

since

$1 \le a, b \le \dfrac{p - 1}{2}, \tag 4$

we have

$2 \le a + b \le 2\dfrac{p - 1}{2} = p - 1; \tag 5$

it follows that

$a + b \not \equiv 0 \mod p, \tag 6$

and hence that $a + b$ is invertible modulo $p$, since

$\Bbb Z_p = \Bbb Z / p\Bbb Z \tag 7$

forms a field, and $p$ is odd, so $2 \not \equiv 0 \mod p$; from this we conclude that

$a - b \equiv 0 \mod p, \tag 8$

or

$a \equiv b \mod p, \tag 9$

contrary to the hypothesis

$a \not \equiv b \mod p; \tag{10}$

therefore,

$a^2 \not \equiv b^2 \mod p, \tag{11}$

$OE\Delta$.


The idea is that $a^2-b^2=(a-b)(a+b)$ and saying $a^2 \not\equiv b^2 (mod p)$ is the same as saying, $(a-b)(a+b) \not\equiv 0 (mod p)$

You do have from your conditions that $a-b \not\equiv 0 (mod p)$ and since $a+b<p$ then ofcourse $a+b \not\equiv 0 (mod p)$ and $p$ is a prime so now you have that, $a^2 \not\equiv b^2 (mod p)$