Transpose of inverse vs inverse of transpose

Is $(A^{-1})^T = (A^T)^{-1}$ you ask.

Well $$\begin{align} A^T(A^{-1})^T = (A^{-1}A)^{T} = I^T = I \\ (A^{-1})^TA^T = (AA^{-1})^{T} = I^T = I \end{align} $$

This proves that the inverse of $A^T$ is $(A^{-1})^T$. So the answer to your question is yes.

Here I have used that $$ A^TB^T = (BA)^T. $$ And we have used that the inverse of a matrix $A$ is exactly (by definition) the matrix $B$ such that $AB = BA = I$.

Given that $A\in \mathbb{R}^{n\times n}$ is invertible, $(A^{-1})^T = (A^T)^{-1}$ holds.

Proof:

$A$ is invertible and $\textrm{rank }A = n = \textrm{rank }A^T,$ so $A^T$ is invertible as well. Conclusion: $$(A^{-1})^T = (A^{-1})^TA^T(A^T)^{-1} = (AA^{-1})^T(A^T)^{-1} = \textrm{id}^T(A^T)^{-1} = (A^T)^{-1}.$$

Suppose $A$ is an invertible square matrix. Then $A$ has a left-inverse; let us just call it $C$ to avoid confusing notation at first. So, $A$ being (left-)invertible means existence of some $C$ so that $$CA=I$$ Taking transposes on both sides and using the rule for transposing a product yields $$A^T C^T = I$$ The last equation shows that $A^T$ has a right-inverse (which is actually $C^T$), and so $A^T$ is (right-)invertible.

Do the analogous thing for the inverses from the other side, or appeal to the fact that for finite-dimensional matrices having an inverse from one side is enough to ensure that the same matrix is a two-sided inverse.

What we have then shown, is that $\left( A^T \right) ^{-1}$ exists and is given by the formula: $$\left( A^T \right) ^{-1} = \left( A^{-1} \right) ^T$$

This also shows, since the transpose of a transpose is the original matrix, that if $A$ is not invertible, then neither is $A^T$.