Of all polygons inscribed in a given circle which one has the maximum sum of squares of side lengths?
Yes, the maximal sum is the one of the equilateral triangle, that is $9R^2$.
Since Prove that in any triangle $ABC$, $\cos^2A+\cos^2B+\cos^2C\geq\frac{3}{4}$ then $$ \sin^2 A+\sin^2 B+\sin^2 C=3-\cos^2 A-\cos^2 B-\cos^2 C\leq \frac{9}{4} $$ where $A$, $B$ and $C$ are non negative numbers such that $A+B+C=\pi$. Hence, for any inscribed triangle, the sum of the squares of the sides is $$(2R\sin A)^2+(2R\sin B)^2+(2R\sin C)^2\leq 9R^2.$$
This problem can be stated as
$$ \max_{n}\sum_{k=1}^n\left(2r\sin\left(\frac{\theta_k}{2}\right)\right)^2 $$
s. t.
$$ \sum_{k=1}^n\theta_k = 2\pi $$
but
$$ \sum_{k=1}^n\left(2r\sin\left(\frac{\theta_k}{2}\right)\right)^2\ge n\left(2^{2n}r^{2n}\prod_{k=1}^n\sin^2\left(\frac{\theta_k}{2}\right)\right)^{\frac 1n} $$
assuming $\theta_1=\cdots=\theta_n$ we have
$$ \sum_{k=1}^n\left(2r\sin\left(\frac{\theta_k}{2}\right)\right)^2\ge n\left(2^{2n}r^{2n}\sin^{2n}\left(\frac{\pi}{n}\right)\right)^{\frac 1n} = n2^2r^2\sin^{2}\left(\frac{\pi}{n}\right) $$
Now calling
$$ f(n) = n\sin^{2}\left(\frac{\pi}{n}\right) $$
we have clearly a maximum about $n = 3$ as can be depicted in the attached plot
Let $\theta_k$ be the successive angles subtended by the sides, but the last one. The sum of squares is given by
$$4\sum_{k=1}^n\sin^2\frac{\theta_k}2+4\sin^2\left(\pi-\frac12\sum_{k=1}^n\theta_k\right)$$ which has the same extrema as $$\sum_{k=1}^n\cos\theta_k-\cos\left(\sum_{k=1}^n\theta_k\right).$$
Canceling the gradient,
$$\sin\theta_k=\sin\left(\sum_{k=1}^n\theta_k\right).$$
This shows that all angles $\theta_k$ must be equal, and then
$$n\cos\theta-\cos n\theta$$ is minimized with $n\theta=2\pi$.
Finally,
$$n\cos\frac{2\pi} n-\cos\pi$$ is the smallest with $n=3$.