When should I take conjugate transpose of a complex matrix, and when transpose of it?

Briefly: "most of the time", the correct analog for extending a matrix expression involving a transpose will be the conjugate-transpose. In the case of the inverse, however, the formula for $A^{-1}$ should be a "nice" (i.e. complex-differentiable) function and so we must use the entry-wise transpose.

---

In the case of real matrices, the transpose "usually" arises when we consider the relationship of a matrix $A$ to the inner-product $\langle x, y \rangle_{\Bbb R} = y^Tx = x^Ty$. More abstractly, this occurs when we consider how the linear transformation induced by $A$ interacts with the usual (Euclidean) geometry on $\Bbb R^n$. For instance, we have the following definitions and statements involving the transpose of a real matrix.

Definitions:

• The length of a vector $x$ (denoted $\|x\|$) is $\sqrt{\langle x,x\rangle_{\Bbb R}}$. Notably, $\langle x,x \rangle_{\Bbb R} > 0$ whenever $x \neq 0$.
• $A$ is symmetric (i.e. self-adjoint) when $A = A^T$, or equivalently when $\langle Ax, y \rangle_{\Bbb R} = \langle x, Ay \rangle_{\Bbb R}$.
• $A$ is orthogonal (i.e. satisfies $\langle Ax, Ay \rangle_{\Bbb R} = \langle x, y \rangle_{\Bbb R}$) when $A^TA = I$
• $A$ is positive definite when ($A$ is symmetric and) for all non-zero $x$, $\langle x,Ax \rangle_{\Bbb R} > 0$

Theorems:

• Cauchy-Schwarz: $\langle x,y \rangle_{\Bbb R} \leq \|x\| \cdot \|y\|$. Moreover, the angle between unit-vectors $x,y$ is $\cos^{-1}(\langle x,y \rangle_{\Bbb R})$.
• For a rectangular $A$, $A^TA$ will have the same rank as $A$, and $\sqrt{\det(A^TA)}$ is the "volume" spanned by the columns of $A$ when $A$ has linearly independent columns.
• The spectral theorem: if $A$ is symmetric, then $A$ is diagonalizable with real eigenvalues. Moreover, $A$ is orthogonally diagonalizable so that $A = UDU^T$ for some diagonal $D$ and orthogonal $U$.
• $A$ is positive definite if and only if it is symmetric with positive eigenvalues. This occurs if and only if $x,y \mapsto \langle Ax, y \rangle_{\Bbb R}$ defines an inner product.
• Every (rectangular) matrix $A$ has a singular-value decomposition $A = U \Sigma V^T$ (equivalent to the spectral theorem).

All of these statements and theorems have analogs when we consider complex matrices over the Hermitian inner product, which is defined by $\langle x,y \rangle = y^*x$. In the complex context, any $A^T$ is replaced with $A^*$, the conjugate-transpose of $A$.

---

Now, let's consider the entry-wise transpose for complex matrices and the corresponding bilinear form $(x,y) = y^Tx = x^Ty$. Here are some things that go wrong.

• It is not true that $\langle x,x \rangle_{\Bbb R} > 0$ whenever $x \neq 0$. For instance, the non-zero vector $x = (1,i) \in \Bbb C^2$ satisfies $(x,x) = 0$.
• Cauchy-Schwarz fails, for instance, with $x = (1,i)$ and $y = (1,-i)$.
• $A^TA$ no longer has the same rank as $A$. For instance, $$ A = \pmatrix{1&i\\i&-1} $$ satisfies $A^TA = 0$
• $A$ can be symmetric without being diagonalizable. Consider for instance the $A$ given above, which fails to be diagonalizable.
• For any matrix $A$, $x^T A x$ is a complex polynomial on the entries of $x$. We can never have $x^TAx > 0$ for all $x \neq 0$.

There is, however, something gained in this case. Because $(x,y)$ is a polynomial on the entries of $x$ and $y$ (whereas $\langle x, y \rangle$ fails to be complex-differentiable), formulas involving the entry-wise transpose behave nicely with respect to computations involving complex numbers, including complex-differentiation.

For instance: the matrices satisfying $A^T = A$ form a complex subspace of $\Bbb C^{n \times n}.$ Also, the set of complex-orthogonal matrices (i.e. matrices satisfying $A^TA = I$) forms a smooth manifold in $\Bbb C^{n \times n}$.

Another consequence of all this, as you said, is that the correct choice for you determinant formula is the entry-wise transpose rather than the conjugate transpose. In this case, the formula for the cofactor matrix bears no relation to the Euclidean geometry on $\Bbb R^n$ or $\Bbb C^n$.

Over any field $k$ a matrix $A\in k^{2\times 2}$ where $\det(A)\neq 0$ the inverse is given by $$\frac{1}{\det(A)}\begin{pmatrix}a_{22}&-a_{12}\\-a_{21}&a_{11}\end{pmatrix}$$ as you can easily calculate: $$\begin{pmatrix}a_{11}&a_{12}\\a_{21}&a_{22}\end{pmatrix}\begin{pmatrix}a_{22}&-a_{12}\\-a_{21}&a_{11}\end{pmatrix}=\begin{pmatrix}\det(A)&0\\0&\det(A)\end{pmatrix}$$

I don't see why we should. We have the well-known formula, valid over any commutative ring: $$A\,^{\mathrm t\mkern-2.5mu}(\operatorname{com}A)=(\det A)I$$ where $\;\operatorname{com}A$ denotes the comatrix of $A$, a.k.a. matrix of cofactors.

This formula results directly from Laplace's formula for the expansion of a determinant.