What does $A^{-1}=A^T$ have to do with “orthogonality”?

$A^{-1} = A^T \iff A^T A = I$, so let's explore what this latter property means. (Assume that we are working in a real vector space, specifically $\mathbb{R}^n$.) Write

$$A = \begin{pmatrix} \uparrow & \uparrow & \dots&\uparrow \\ v_1 &v_2&\dots&v_n \\ \downarrow&\downarrow&\dots&\downarrow \end{pmatrix} $$ then $$A^TA = \begin{pmatrix}\leftarrow & v_1 & \rightarrow \\ \leftarrow & v_2 & \rightarrow \\ \vdots & \vdots & \vdots \\ \leftarrow & v_n & \rightarrow \end{pmatrix} \begin{pmatrix} \uparrow & \uparrow & \dots&\uparrow \\ v_1 &v_2&\dots&v_n \\ \downarrow&\downarrow&\dots&\downarrow \end{pmatrix} =\begin{pmatrix} v_1\cdot v_1 & v_1\cdot v_2&\dots&v_1 \cdot v_n \\ v_2\cdot v_1 & v_2\cdot v_2&\dots&v_2 \cdot v_n \\ \vdots & \vdots & \ddots & \vdots \\ v_n\cdot v_1 & v_n\cdot v_2&\dots&v_n \cdot v_n \\ \end{pmatrix}. $$ So, $A^T A = I$ corresponds exactly to $v_i \cdot v_i = 1$ and $v_i \cdot v_j = 0$ for $i \neq j$, i.e. $\{v_i\}$ are an orthonormal system. So, if a matrix is orthogonal then its columns are orthogonal. Similarly, you can see that its rows are also orthogonal, since if $A$ is orthogonal then $A^T$ is also.

Beacuase a matrix has that property if and only if all columns are orthonormal. This is also equivalent to the assertion that all rows are orthonormal.

When this happens, the determinant is $1$ or $-1$, which you can verify explicitly. Not only is the determinant a unit, the transformation preserves orthogonal vectors.

Let $u,v$ be orthogonal vectors. Then the inner product $\langle Au , Av\rangle = \langle u, A^TAv \rangle$. This is true because of the nature of the inner product; I encourage you to explicitly verify this. However, $A^TA = I$, so $$\langle Au, Av \rangle = \langle u, v \rangle$$ Thus, orthogonality has been preserved. This is the real reason why we need $A^T = A^{-1}$ for a matrix to be orthogonal.