Matrix-convexity of inverse of the cofactor matrix

Not just $3\times 3$, but in general, the map $A \mapsto \det(A^{-1})A$ is operator convex on positive definite matrices.

Proof sketch. $\newcommand{\pfrac}[2]{\left(\tfrac{#1}{#2}\right)}$ If suffices to prove the following matrix inequality for two psd matrices $A, B$: \begin{equation*} \frac{A+B}{\det\pfrac{A+B}{2}} \le \frac{A}{\det{A}} + \frac{B}{\det B}. \end{equation*} Since there exists an invertible matrix $P$ such that $P^*AP=I$ and $P^*BP=D$, where $D$ is a positive and diagonal, we may equivalently show that \begin{equation*} \frac{I+D}{\det\pfrac{I+D}{2}} \le I + \frac{D}{\det D}. \end{equation*} Proving this inequality reduces to showing $n$ inequalities of the form (clearly, it suffices to show one): \begin{equation*} \frac{1+d_1}{\prod_{i=1}^n\pfrac{1+d_i}{2}} \le 1 + \frac{d_1}{\prod_{i=1}^n d_i}\quad\equiv\quad 2 \le \pfrac{1+d_2}{2}\cdots\pfrac{1+d_n}{2} + \pfrac{1+d_2}{2d_2}\cdots\pfrac{1+d_n}{2d_n}. \end{equation*} The latter inequality can be shown by induction, so am omitting typing the routine calculations.

End of proof (by Denis Serre). You have to prove $$2^n\le\prod_2^n(1+d_i)+\prod_2^n(1+\frac1{d_i}).$$ Just write the right-hand side as a sum over all elementary monomials (monomials whose partial degrees are $0$ or $1$): $$\sum_m\left(m(d)+\frac1{m(d)}\right).$$ There are $2^{n-1}$ such monomials, and each sum $m+\frac1m$ is $\ge2$.

Well, this is not an answer. But I cannot resist to mention the following equivalent property. Let $A\mapsto \hat A$ denote the cofactor map, and $B\mapsto \check B$ its inverse. For positive definite symmetric matrices, $\hat A=(\det A)A^{-1}$ and $\check B=(\det B)^{\frac1{n-1}}B^{-1}$. Then the harmonic mean is less than or equal to the cofactor mean over the cone of positive definite matrices: $$\langle A^{-1}\rangle^{-1}\le\widehat{\langle\hat A\rangle}.$$