$A$ is invertible if and only if $A^t$ is invertible

If there is a $B$, such that $AB=I$, then $B^tA^t=I$ and by uniqueness of inverse you have $B^t$ as inverse.

You perform Gaussian elimination and it succeeds. This shows that the row rank is equal to the column rank. A square matrix is invertible iff it has maximal rank.

The answer from Arash uses $B^tA^t=(AB)^t$ to prove that if a square matrix $A$ is invertible, then $A^t$ is invertible: $(A^{-1})^tA^t=(AA^{-1})^t = I^t=I$, so $A^t$ is invertible by the invertible matrix theorem. This also shows that $(A^t)^{-1}=(A^{-1})^t$.

Here is a proof of the other direction, since the question asks for "if and only if":

Suppose $A^t$ is invertible. Then

$$ \begin{aligned} \left(\left( (A^t)^{-1}\right)^tA\right)^t &= A^t\left(\left( (A^t)^{-1}\right)^t\right)^t \\ &= A^t(A^t)^{-1} \\ &= I \end{aligned} $$

(using $(A^t)^t=A$, which we now use again). We have shown

$$\left(\left( (A^t)^{-1}\right)^tA\right)^t=I$$

so

$$\left((A^t)^{-1}\right)^tA = \left(\left(\left( (A^t)^{-1}\right)^tA\right)^t \right)^t = I^t = I$$

so $A$ is invertible.