Maximal determinant of $3 \times 3$ matrix where sum of squared entries is $\leq 1$

Disclaimer : This is not a proof ; just important tracks :

This maximum determinant looks to be $\dfrac{1}{3 \sqrt{3}}$.

It is obtained for diagonal matrices (as you mention it) such as

$$\begin{pmatrix}\tfrac{1}{\sqrt{3}} & 0 & 0\\0 & \tfrac{1}{\sqrt{3}} & 0\\0 & 0 & \tfrac{1}{\sqrt{3}} \end{pmatrix} \ \text{or} \ \begin{pmatrix}-\tfrac{1}{\sqrt{3}} & 0 & 0\\0 & -\tfrac{1}{\sqrt{3}} & 0\\0 & 0 & \tfrac{1}{\sqrt{3}} \end{pmatrix} \ \text{or...} \tag{1}$$

but also, much more generally, by taking any special orthogonal matrix $A$ (corresponding to columns arranged as a direct basis in $\mathbb{R}^3$) and dividing all its entries by $\sqrt{3}$.

Explanation : A direct orthogonal matrix $A$ has determinant $1$ ; each one of its columns has norm $1$, i.e. $\sum_i a_{ij}^2=1$ for every $j$. Thus

$$\sum_j \sum_i a_{ij}^2=1+1+1=3.$$

The result follows because if one divides such a matrix by $k=\sqrt{3}$, its determinant is divided by $k^3=3\sqrt{3}$.

Conjecture (once again : not a proof) : orthogonal matrices divided by $\sqrt{3}$ look to be/are the only solutions to the issue. In support of this hypothesis, orthogonal matrices, for given norms of their columns, maximize their determinant (Hadamard's inequality : https://en.wikipedia.org/wiki/Hadamard%27s_inequality).

Remark : The square root of the sum of squared entries $a_{ik}^2$ is the Frobenius norm $\|A\|_F$ of matrix $A$ (http://mlwiki.org/index.php/Frobenius_Norm). It has many properties. In particular, it is invariant by left or right multiplication by an orthogonal matrix.

Therefore, one could recast the issue into the following one : find the maximum of function $\det$ on the unit ball for $\|.\|_F$.

This determinant measures the volume of the parallelepiped built on the column vectors. If you keep the vector lengths constant, which keeps the sum of squares constant, the volume is maximized when the vectors are orthogonal.

Now let the lengths be $a,b,c$, you want to maximize

$$V=abc$$ under the constraint $$a^2+b^2+c^2=1.$$

By Lagrangian multipliers,

$$abc+\lambda(a^2+b^2+c^2-1)$$

$$\begin{cases}bc+2\lambda a&=0, \\ac+2\lambda b&=0, \\ab+2\lambda c&=0, \\a^2+b^2+c^2-1&=0\end{cases}$$

Eliminating $\lambda$ from the three first equations, you get $a^2=b^2=c^2$, and using the last, $a^2=b^2=c^2=\dfrac13$. Hence

$$\Delta=\frac1{\sqrt{27}}.$$

---

Direct vector proof:

Let $\vec a,\vec b,\vec c$ be the column vectors of the matrix. The determinant is the mixed product

$$\Delta=\vec a\cdot(\vec b\times\vec c).$$

Using a Lagrangian multiplier, we maximize

$$\vec a\cdot(\vec b\times\vec c)+\lambda(\vec a^2+\vec b^2+\vec c^2-1).$$

Taking the gradient yields

$$\begin{cases}\vec b\times\vec c+2\lambda\vec a=0, \\\vec c\times\vec a+2\lambda\vec b=0, \\\vec a\times\vec b+2\lambda\vec c=0, \\\vec a^2+\vec b^2+\vec c^2-1=0. \end{cases}$$

Then applying dot products on the first equations, we get

$$\lambda\vec a\vec b=\lambda\vec b\vec c=\lambda\vec c\vec a=0,$$ which shows that the vectors must be orthogonal, and we are back to the above scalar problem.

Instead of the inequality constraint $\| \mathrm X \|_{\text F}^2 = \mbox{tr} \left( \mathrm X^\top \mathrm X \right)\leq 1$, let us consider the equality constraint $\| \mathrm X \|_{\text F}^2 = 1$. Hence, we have the following maximization problem

$$\begin{array}{ll} \text{maximize} & \det (\mathrm X)\\ \text{subject to} & \mbox{tr} \left( \mathrm X^\top \mathrm X \right) = 1\end{array}$$

Let us use the following Lagrangian

$$\mathcal L (\mathrm X, \mu) := \det (\mathrm X) - \frac{\mu}{2} \left( \mbox{tr} \left( \mathrm X^\top \mathrm X \right) - 1 \right)$$

Differentiating $\mathcal L$ with respect to $\rm X$ and $\mu$ and finding where the derivatives vanish, we obtain

$$\begin{aligned} \det (\mathrm X) \, \mathrm X^{-\top} &= \mu \mathrm X\\ \mbox{tr} \left( \mathrm X^\top \mathrm X \right) &= 1\end{aligned}$$

where Jacobi's formula was used. Left-multiplying both sides of the 1st equation by $\mathrm X^\top$,

$$\begin{aligned} \det (\mathrm X) \, \mathrm I_3 &= \mu \mathrm X^\top \mathrm X\\ \mbox{tr} \left( \mathrm X^\top \mathrm X \right) &= 1\end{aligned}$$

Taking the trace of both sides of the 1st equation, we obtain $\mu = 3 \det (\mathrm X)$ and, thus,

$$\begin{aligned} \det (\mathrm X) \, \mathrm I_3 &= 3 \det (\mathrm X) \, \mathrm X^\top \mathrm X\\ \mbox{tr} \left( \mathrm X^\top \mathrm X \right) &= 1\end{aligned}$$

Assuming (again) that $\mathrm X$ is invertible, both sides of the 1st equation can be divided by $\det (\mathrm X)$,

$$\begin{aligned} \mathrm X^\top \mathrm X &= \frac 13 \mathrm I_3\\ \mbox{tr} \left( \mathrm X^\top \mathrm X \right) &= 1\end{aligned}$$

Thus, if $\Omega$ is a $3 \times 3$ special orthogonal matrix,

$$\bar{\mathrm X} := \color{blue}{\frac{1}{\sqrt 3} \Omega}$$

yields the maximal determinant, $\det (\bar{\mathrm X}) = \dfrac{1}{3 \sqrt 3}$, subject to the equality constraint $\| \mathrm X \|_{\text F}^2 = 1$.