An inequality using convex functions: If $\frac{a^2}{1+a^2}+\frac{b^2}{1+b^2}+\frac{c^2}{1+c^2} = 1$ then $abc \le \frac1{2\sqrt 2}$

One possibility is to denote $A=\frac{a^2}{1+a^2}$, etc., so $a^2=\frac{A}{1-A}$, etc. We are given $A+B+C=1$ and want to prove that $8ABC\le(1-A)(1-B)(1-C)$, or, after opening the parentheses, replacing the sum by $1$, and putting all $ABC$ on the left, $$ 9ABC\le AB+BC+AC=(AB+BC+AC)(A+B+C)\,. $$ Now just use AM-GM for each expression in parentheses on the right separately.


Variational Analysis Approach

If $$ \frac{a^2}{1+a^2}+\frac{b^2}{1+b^2}+\frac{c^2}{1+c^2}=1\tag{1} $$ then variations of $a$, $b$, and $c$ must satisfy $$ \frac{2a\,\delta a}{\left(1+a^2\right)^2}+\frac{2b\,\delta b}{\left(1+b^2\right)^2}+\frac{2c\,\delta c}{\left(1+c^2\right)^2}=0\tag{2} $$ To maximize $abc$, for each $\delta a$, $\delta b$, and $\delta c$ that satisfy $(2)$, we must have $$ \frac{\delta a}a+\frac{\delta b}b+\frac{\delta c}c=0\tag{3} $$ Orthogonality with $(2)$ and $(3)$ means that there is a $\lambda$ so that $$ \frac{a^2}{\left(1+a^2\right)^2}=\frac{b^2}{\left(1+b^2\right)^2}=\frac{c^2}{\left(1+c^2\right)^2}=\lambda\tag{4} $$ Since $\frac{x^2}{\left(1+x^2\right)^2}$ is two-to-one, for each value of $\lambda$, there are two possible values of $a$, $b$, or $c$. Using Vieta's Formulas, the product of the two solutions of $\frac{x^2}{\left(1+x^2\right)^2}=\lambda$ is $1$.

If it is not the case that $a=b=c$, then suppose $ab=1$. Then $$ \begin{align} \frac{a^2}{1+a^2}+\frac{b^2}{1+b^2} &=\frac{a^2}{1+a^2}+\frac{1/a^2}{1+1/a^2}\\ &=\frac{a^2}{1+a^2}+\frac1{1+a^2}\\[6pt] &=1\tag{5} \end{align} $$ $(1)$ and $(5)$ imply that $c=0$. Therefore, we must have $a=b=c$. That is, $$ \frac{a^2}{1+a^2}=\frac{b^2}{1+b^2}=\frac{c^2}{1+c^2}=\frac13\tag{6} $$ which means $a=b=c=\frac1{\sqrt2}$. Therefore, $abc=\frac1{2\sqrt2}$ at the maximum. That is, $$ abc\le\frac1{2\sqrt2}\tag{7} $$


Convex Analysis Approach

We are given that $$ \frac{a^2}{1+a^2}+\frac{b^2}{1+b^2}+\frac{c^2}{1+c^2}=1\tag{8} $$ Case 1: $a,b,c\le1$

Using the convex function on $t\le0$, $u(t)=\frac{e^t}{1+e^t}$, we have $$ \begin{align} \frac{(abc)^{2/3}}{1+(abc)^{2/3}} &=\frac{e^{\frac13\left(\log\left(a^2\right)+\log\left(b^2\right)+\log\left(c^2\right)\right)}}{1+e^{\frac13\left(\log\left(a^2\right)+\log\left(b^2\right)+\log\left(c^2\right)\right)}}\\ &\le\frac13\left(\frac{a^2}{1+a^2}+\frac{b^2}{1+b^2}+\frac{c^2}{1+c^2}\right)\\ &=\frac13\tag{9} \end{align} $$ Applying $\frac x{1-x}$ to $(9)$ and raising to the $\frac32$ power gives $$ abc\le\frac1{2\sqrt2}\tag{10} $$ Case 2: One of $a$, $b$, or $c$ is greater than $1$. Without loss of generality, assume $a\gt1$.

Since $a\gt1$, we have that $b,c\lt1$. Now we have $$ \begin{align} \frac{bc}{1+bc} &=\frac{e^{\frac12\left(\log\left(b^2\right)+\log\left(c^2\right)\right)}}{1+e^{\frac12\left(\log\left(b^2\right)+\log\left(c^2\right)\right)}}\\ &\le\frac12\left(\frac{b^2}{1+b^2}+\frac{c^2}{1+c^2}\right)\\ &=\frac12\left(1-\frac{a^2}{1+a^2}\right)\tag{11} \end{align} $$ Applying $\frac x{1-x}$ to $(11)$ and multiplying by $a$ gives $$ \begin{align} abc &\le a\frac{\frac12\left(1-\frac{a^2}{1+a^2}\right)}{\frac12\left(1+\frac{a^2}{1+a^2}\right)}\\ &=\frac a{1+2a^2}\\ &=\frac a{\left(1-\sqrt2a\right)^2+2\sqrt2a}\\ &\le\frac1{2\sqrt2}\tag{12} \end{align} $$


I believe I can get you started at least. Break down your equation a little.

$a,b,c \in \mathbb{R}^+ \rightarrow \frac{1}{1+n^2} \in [0,1],\;\;\; n = a,b,c$

This means that your original equation could be considered a convex combination as follows: $a^2\lambda_1 + b^2\lambda_2 +c^2\lambda_3 = 1$, where the $\lambda$ values are the fractions defined above.

That, however, is just part of the setup. Basically you've determined that $\frac{1}{1+a^2} + \frac{1}{1+b^2} + \frac{1}{1+c^2} = 1$, by definition of convex combination. My interpretation is that what you're really being asked to prove is what is the minimum value that each $\lambda_i$ can take in a convex combination. This is because, given the definition of $\lambda_i$ for this problem, when a $\lambda$ reaches a minimum, it is because the variable ($a,b,c$ for a given $\lambda$) will be at a maximum. When you maximize $a,b,$ and $c$ then you also maximize their product $abc$.

Consider the case with just two variables between 0 and 1 that must also sum to 1. You want to minimize them both, so the only possible solution is that they equal each other. Neither can be improved without departing from the constraint of the problem (namely that they sum to 1). This extends quite easily to 3 variables, and there are plenty of proofs to look up online.

Back to the specifics of your problem: Since all of the coefficient expressions are equivalent, you can assume $a = b = c$ at the extreme value. Knowing this:

$\frac{3a^2}{1+a^2} = 1 \rightarrow a = \frac{1}{\sqrt{2}}$ and $abc = (\frac{1}{\sqrt{2}})^3 = \frac{1}{2\sqrt{2}} $