Hard inequality :$\Big(\frac{1}{a^2+b^2}\Big)^2+\Big(\frac{1}{b^2+c^2}\Big)^2+\Big(\frac{1}{c^2+a^2}\Big)^2\geq \frac{3}{4}$

Nelson Faustino guessed that $(a^2+b^2)(b^2+c^2)(c^2+a^2)\le 8$. Let us prove it.

Proof: We will use the following lemma whose proof is given later.

Lemma 1: Let $a, b, c\ge 0$ with $11(a^2+b^2+c^2) + 4(a+b+c) - 45 \le 0$. Then $(a^2+b^2)(b^2+c^2)(c^2+a^2)\le 8$.

Let us begin. Using AM-GM, we have $$\frac{1}{(a^2+b^2)^2} + \frac{1}{(b^2+c^2)^2} + \frac{1}{(c^2+a^2)^2} \ge 3\sqrt[3]{\frac{1}{(a^2+b^2)^2(b^2+c^2)^2(c^2+a^2)^2}}.$$ It suffices to prove that $(a^2+b^2)(b^2+c^2)(c^2+a^2)\le 8$. To this end, we have \begin{align} &a, b, c > 0, \quad a^ab^bc^c=1 \qquad\qquad (1)\\ \Longrightarrow \quad & a\ln a + b\ln b + c\ln c = 0, \quad 0 < a, b, c < \frac{159}{100}\qquad\qquad (2)\\ \Longrightarrow \quad &11(a^2+b^2+c^2) + 4(a+b+c) - 45 \le 0, \quad a, b, c > 0. \qquad\qquad (3) \end{align} Here, in (2), since $-u\ln u \le \frac{1}{\mathrm{e}}$ for $u>0$, we have $a\ln a, b\ln b, c\ln c \le \frac{2}{\mathrm{e}}$ which, when combined with $\frac{159}{100}\ln \frac{159}{100} > \frac{2}{\mathrm{e}}$, results in $a, b, c < \frac{159}{100}$ (noting that $x \mapsto x\ln x$ is strictly increasing for $x > 1$); In (3), we have used the fact that $$x\ln(x) \ge \frac{11}{26}x^2 + \frac{2}{13}x - \frac{15}{26}, \quad \forall 0 < x < \frac{159}{100}.\qquad (4)$$ Then, according to Lemma 1, the desired result follows. We are done.

Proof of (4): It suffices to prove that $$\ln x \ge \frac{\frac{11}{26}x^2 + \frac{2}{13}x - \frac{15}{26}}{x}, \quad \forall 0 < x < \frac{159}{100}.$$ Let $h(x) = \ln x - \frac{\frac{11}{26}x^2 + \frac{2}{13}x - \frac{15}{26}}{x}$. We have $h'(x) = -\frac{11(x-1)(x-\frac{15}{11})}{26x^2}$. Thus, $h(x)$ is decreasing on $(0, 1)$, increasing on $(1, \frac{15}{11})$ and decreasing on $(\frac{15}{11}, \infty)$. Note that $h(1) = 0$ and $h(\frac{159}{100}) > 0 $. The desired result follows.

Proof of Lemma 1: One of the methods is the uvw method as follows. However, I hope to see nice proofs of Lemma 1.

Let $a+b+c = 3u$, $ab+bc+ca = 3v^2$ and $abc = w^3$.

The constraint $11(a^2+b^2+c^2) + 4(a+b+c) - 45 \le 0$ becomes $11(9u^2 - 6v^2) + 12u - 45 \le 0$. The constraint does not involve $w^3$.

We need to prove that $f(w^3) = 8 - (a^2+b^2)(b^2+c^2)(c^2+a^2) = w^6+6u(9u^2-6v^2)w^3-81u^2v^4+54v^6+8 \ge 0$. Since $9u^2 - 6v^2 \ge 0$ and $w^3\ge 0$, $f(w^3)$ is increasing with $w^3$. Thus, we only need to prove the case when $a=b$ or $c=0$. The rest is not hard and thus omitted.

Define: $$F(a,b,c)=\left(\frac{1}{a^2+b^2}\right)^2+\left(\frac{1}{b^2+c^2}\right)^2+\left(\frac{1}{c^2+a^2}\right)^2$$

By the method of Lagrange multipliers, the minimizer of $F$ under the constraint $a^ab^bc^c=1$ must be a critical point of the Lagrangian

$$L(a,b,c,\lambda)=\left(\frac{1}{a^2+b^2}\right)^2+\left(\frac{1}{b^2+c^2}\right)^2+\left(\frac{1}{c^2+a^2}\right)^2+\lambda(a^ab^bc^c-1)$$

To find the critical points we solve $\nabla L=0$:

$$\frac{\partial L}{\partial\lambda}=a^ab^bc^c-1=0$$ $$\frac{\partial L}{\partial a}=-\frac{4a}{(a^2+b^2)^3}-\frac{4a}{(a^2+c^2)^3}+\lambda(a^ab^bc^c)(1+\log a)=0$$

and similarly find the equation for $\frac{\partial L}{\partial b}=0$ and $\frac{\partial L}{\partial c}=0$. We got a system of 4 (non-linear) equations with 4 variables.

Let $a,b,c,\lambda$ be a solution. Since $a^ab^bc^c=1$, at least one of $a,b,c$ is $\leq 1$ and at least one is $\geq 1$, so WLOG we can treat two cases: $a\leq b\leq 1\leq c$ and $a\leq 1\leq b\leq c$ .

• First case $a\leq b\leq 1\leq c$: Isolating $\lambda$ in the equations $\frac{\partial L}{\partial a}=0$ and $\frac{\partial L}{\partial b}=0$ leads to $$\lambda=\left[\frac{1}{(a^2+b^2)^3}+\frac{1}{(a^2+c^2)^3}\right]\cdot\frac{a}{1+\log a}=\left[\frac{1}{(a^2+b^2)^3}+\frac{1}{(b^2+c^2)^3}\right]\cdot\frac{b}{1+\log b}$$ If $a<b$ then $$\left[\frac{1}{(a^2+b^2)^3}+\frac{1}{(a^2+c^2)^3}\right]>\left[\frac{1}{(a^2+b^2)^3}+\frac{1}{(b^2+c^2)^3}\right]$$ and $$\frac{a}{1+\log a}>\frac{b}{1+\log b}$$ by contradiction to the equality above. Therefore we conclude $a=b$, and we may reduce our problem to 2 dimensions: Minimize $$F(a,a,c)=\frac{1}{4a^4}+2\left(\frac{1}{a^2+c^2}\right)^2$$ Among all $a,c$ s.t. $a^{2a}c^c=1$. Using the square super root function we put $c=\text{ssrt}(a^{-2a})$ in the above expression and get $$G(a)=\frac{1}{4a^4}+2\left(\frac{1}{a^2+\text{ssrt}(a^{-2a})^2}\right)^2$$ Convince yourself that $G$ attains its minimum at $a=1$ (I used a computer, maybe possible to do it analytically), so to minimize $G$ we must take $a=1$. We conclude $a=b=1$, and then $c=\text{ssrt}(1)=1$ as well.

• Second case $a\leq 1\leq b\leq c$: Isolating $\lambda$ in the equations $\frac{\partial L}{\partial b}=0$ and $\frac{\partial L}{\partial c}=0$ leads to $$\lambda=\left[\frac{1}{(b^2+a^2)^3}+\frac{1}{(b^2+c^2)^3}\right]\cdot\frac{b}{1+\log b}=\left[\frac{1}{(c^2+a^2)^3}+\frac{1}{(c^2+b^2)^3}\right]\cdot\frac{c}{1+\log c}$$ If $b<c$ then $$\frac{1}{(b^2+a^2)^3}\frac{b}{1+\log b}>\frac{1}{(c^2+a^2)^3}\frac{c}{1+\log c}$$ and $$\frac{1}{(b^2+c^2)^3}\frac{b}{1+\log b}>\frac{1}{(c^2+b^2)^3}\frac{c}{1+\log c}$$ by contradiction to the equality above. Therefore we conclude $b=c$, and continue as in the first case to show $a=b=c=1$.

To conclude: $$\min\{F(a,b,c);a^ab^bc^c=1\}=F(1,1,1)=\frac{3}{4}$$

Remark: I may have over-complicated things. If you can show in a more simple way that $a=b=c=\lambda=1$ is the only solution to $\nabla L=0$ then you are done.