$3\geq\sum\limits_{cyc}\frac{(x+y)^{2}x^{2}}{(x^{2}+y^{2})^{2}}$ with $x,y,z >0$

Remark: Let us describe simply a known trick for the problem of proving $f(u)+f(v)+f(w)\ge 0$ under constraint $uvw=1$ and $u, v, w > 0$.

The method of Lagrange multipliers yields the system of equations \begin{align} f'(u) &= \lambda vw, \\ f'(v) &= \lambda uw, \\ f'(w) &= \lambda uv,\\ uvw &= 1.\tag{1} \end{align} Clearly, we have $uf'(u) = vf'(v) = wf'(w) = \lambda$. If the equation $xf'(x) = c$ has at most two distinct positive real solutions for any $c \in \mathbb{R}$, then two of $u, v, w$ are equal.

This trick is useful for many problems. For example,

Example 1: Let $a, b, c$ be positive real numbers such that $abc=1$. Prove that $$\frac{a}{a^{11}+1} + \frac{b}{b^{11} + 1} + \frac{c}{c^{11}+1} \le \frac{3}{2}.$$

Example 2: Let $a, b, c$ be positive real numbers such that $abc=1$. Prove that $$\frac{7-6a}{2+a^2} + \frac{7-2b}{2+b^2} + \frac{7-2c}{2+c^2} \ge 1.$$

Example 3: Let $a, b, c$ be positive real numbers such that $abc=1$. Prove that $$\frac{1}{a+3} + \frac{1}{b+3} + \frac{1}{c+3} \ge \frac{a}{a^2+3} + \frac{b}{b^2+3} + \frac{c}{c^2+3}.$$

Example 4: Let $a, b, c$ be positive real numbers such that $abc=1$. Prove that $$\frac{1}{a+4} + \frac{1}{b+4} + \frac{1}{c+4} \ge \frac{a}{a^2+4} + \frac{b}{b^2+4} + \frac{c}{c^2+4}.$$

Example 5: Let $a, b, c$ be positive real numbers such that $abc=1$. Prove that $$\sum_{\mathrm{cyc}} \sqrt{\frac{a}{a+8}} \ge 1.$$

$\phantom{2}$

Use this trick for the OP:

Equivalent problem (as @Display name pointed out): Let $u, v, w > 0$ with $uvw=1$. Prove that $$\frac{(1+u)^2}{(1+u^2)^2}+\frac{(1+v)^2}{(1+v^2)^2}+\frac{(1+w)^2}{(1+w^2)^2}\leq 3.$$

Let $f(x) = 1 - \frac{(1+x)^2}{(1+x^2)^2}$. We need to prove that $f(u)+f(v)+f(w)\ge 0$. The method of Lagrange multipliers yields the system of equations \begin{align} f'(u) &= \lambda vw, \\ f'(v) &= \lambda uw, \\ f'(w) &= \lambda uv, \\ uvw &= 1. \end{align} Clearly, we have $uf'(u) = vf'(v) = wf'(w) = \lambda$. Let us prove that if $(u, v, w, \lambda)$ with $u, v, w > 0$ satisfies the above system of equations, then two of $u, v, w$ are equal.

Let $F(x) = xf'(x) = \frac{2x(1+x)(x^2+2x-1)}{(x^2+1)^3}$.

Clearly, $F(0) = 0$, $F(\sqrt{2}-1) = 0$, $F(x) < 0$ on $(0, \sqrt{2}-1)$, and $F(x) > 0$ on $(\sqrt{2}-1, +\infty)$.

We have $F'(x) = -\frac{2(2x^5+9x^4-14x^2-2x+1)}{(x^2+1)^4}$. Let $G(x) = 2x^5+9x^4-14x^2-2x+1$. From Descartes' sign rule, since there are two sign changes, $G(x) = 0$ has at most two positive real roots. Also, we have $G(0) > 0$, $G(\sqrt{2}-1) = 32-24\sqrt{2} < 0$ and $G(+\infty) = +\infty$. Thus, $G(x) = 0$ has exactly one real solution on $(0, \sqrt{2}-1)$ and $(\sqrt{2}-1, +\infty)$, respectively. Thus, $F'(x) = 0$ has exactly one real solution on $(0, \sqrt{2}-1)$ and $(\sqrt{2}-1, +\infty)$, respectively.

Figure of $F(x)$:

Thus, $F(x) = c$ has at most two distinct positive real solutions for any real number $c$. Since $F(u) = F(v) = F(w)$, we know that two of $u, v, w$ are equal.

For $u = v > 0$ and $w = \frac{1}{u^2}$, it is easy to prove that \begin{align} &f(u) + f(u) + f(\frac{1}{u^2})\\ =\ & \frac{(2u^{10}+4u^9+6u^8+4u^7+3u^6+2u^5+5u^4-u^2-2u+1)(u-1)^2}{(u^2+1)^2(u^4+1)^2}\\ \ge & 0. \end{align} Thus, the inequality is true for $u, v, w > 0$ satisfying the system of equations (1).

It remains to prove that the inequality is true if $\min(u, v, w) \to 0^{+}$ (meaning $(u,v,w)$ approaches the boundary of the constraint).

Let $H(x) = \frac{(1+x)^2}{(1+x^2)^2}$. It is easy to prove that $H(x) \le \frac{147}{100}$ for all real numbers $x$. Note also that $H(x) \le \frac{3}{100}$ for $x \ge 10$. Thus, if $\min(u, v, w) \to 0^{+}$, then $H(u) + H(v) + H(w) \le \frac{147}{100} + \frac{147}{100} + \frac{3}{100} = \frac{297}{100}$. The desired result follows.

We are done.

Partial answer

Following an idea of Display name we have to show :

Let $u,v,w>0$ such that $uvw=1$ then we have : $$\frac{(1+u)^2}{(1+u^2)^2}+\frac{(1+v)^2}{(1+v^2)^2}+\frac{(1+w)^2}{(1+w^2)^2}\leq 3$$

The main idea is to use trigonometry :

Let $u=\tan(\frac{x}{2})$ and $v=\tan(\frac{y}{2})$ and $w=\tan(\frac{z}{2})$

The inequality becomes :

$$\frac{(1+\tan(\frac{x}{2}))^2}{(1+\tan^2(\frac{x}{2}))^2}+\frac{(1+\tan(\frac{y}{2}))^2}{(1+\tan^2(\frac{y}{2}))^2}+\frac{(1+\tan(\frac{z}{2}))^2}{(1+\tan^2(\frac{w}{2}))^2}\leq 3$$

But we have the following relation putting $t=\tan(\frac{x}{2})$ (Weierstrass substitution):

$$\sin(x)=\frac{2t}{1+t^2}$$

$$\cos(x)+1=\frac{2}{1+t^2}$$

So we have :

$$\frac{\cos(x)+\sin(x)+1}{2}=\frac{1+t}{1+t^2}$$

Putting this in the inequality we have to show :

$$\Big(\frac{\cos(x)+\sin(x)+1}{2}\Big)^2+\Big(\frac{\cos(y)+\sin(y)+1}{2}\Big)^2+\Big(\frac{\cos(z)+\sin(z)+1}{2}\Big)^2\leq 3$$

We study the second derivative of the function :

$$f(x)=\Big(\frac{\cos(x)+\sin(x)+1}{2}\Big)^2$$

Which is equal to :

$$f''(x)=-\frac{\sin(x)}{2} - \frac{\cos(x)}{2} - 2 \sin(x) \cos(x)$$ The function $f(x)$ is concave on $[0,p]$ where $p$ have the value :

$$p = 2 \Big(- \tan^{-1}\Big(\frac{3}{2} - \frac{\sqrt{17}}{2} - \sqrt{0.5 (5 - \sqrt{17})}\Big)\Big)>\frac{\pi}{2}$$

So we can apply Jensen's inequality for $x,y,z\in [0,p]$ we have :

$$\sum_{cyc}\Big(\frac{\cos(x)+\sin(x)+1}{2}\Big)^2\leq 3\Big(\frac{\cos(\frac{x+y+z}{3})+\sin(\frac{x+y+z}{3})+1}{2}\Big)^2$$

Or :

$$\sum_{cyc}\Big(\frac{\cos(x)+\sin(x)+1}{2}\Big)^2\leq 3\frac{(1+\tan(\frac{x+y+z}{6}))^2}{(1+\tan^2(\frac{x+y+z}{6}))^2}$$

With the conditions : $0<x<\pi$ and $0<y<\pi$ and $0<z<\pi$ and $\tan(\frac{x}{2})\tan(\frac{y}{2})\tan(\frac{z}{2})=1$

Second edit :

As pointed out by River Li I add a restriction we need to have :

$\frac{3\pi}{4}\leq \frac{x+y+z}{2}$ with the condition $\tan(\frac{x}{2})\tan(\frac{y}{2})\tan(\frac{z}{2})=1$

Or :

$$\frac{3\pi}{4}\leq\tan^{-1}(a)+\tan^{-1}(b)+\tan^{-1}(c) $$

With $a=\tan(\frac{x}{2})$ and $b=\tan(\frac{y}{2})$ and $c=\tan(\frac{z}{2})$

Now if $\max(a,b,c)=a$ and $\min(a,b,c)=c$ we add the restriction $ab>1$ and we have $\frac{a+b}{1-ab}c<0<1$

So we have :

$$\tan^{-1}(a)+\tan^{-1}(b)=\tan^{-1}\Big(\frac{a+b}{1-ab}\Big)+\pi$$

And :

$$\tan^{-1}(a)+\tan^{-1}(b)+\tan^{-1}(c)=\tan^{-1}\Big(\frac{a+b+c-abc}{1-ab-bc-ca}\Big)+\pi$$

So we need to have :

$$\frac{3\pi}{4}\leq\tan^{-1}\Big(\frac{a+b+c-abc}{1-ab-bc-ca}\Big)+\pi$$

Or :

$$\frac{-\pi}{4}\leq\tan^{-1}\Big(\frac{a+b+c-1}{1-ab-bc-ca}\Big)$$

Or :

$$-1\leq \frac{a+b+c-1}{1-ab-bc-ca}$$

Or :

$$1\geq \frac{1-(a+b+c)}{1-ab-bc-ca}$$ Or :

$$1-(a+b+c)\geq 1-ab-bc-ca$$

Or

$$(a+b+c)\leq ab+bc+ca$$

End of the second edit .

So we have :$\frac{3\pi}{2}\leq x+y+z\leq2\pi$ or $\frac{3\pi}{12}\leq\frac{x+y+z}{6}\leq\frac{\pi}{3}$

So $1\leq\tan(\frac{x+y+z}{6})$

But the function $g(x)=\frac{(1+x)^2}{(1+x^2)^2}$ is decreasing on $[1,\infty]$

So $$\sum_{cyc}\Big(\frac{\cos(x)+\sin(x)+1}{2}\Big)^2\leq3\frac{(1+\tan(\frac{x+y+z}{6}))^2}{(1+\tan^2(\frac{x+y+z}{6}))^2}\leq 3$$

And we are done .

Edit:

Since $$(\frac{\sin(x)+\cos(x)+1}{2})^2=0.5+\frac{\sin(x)+\cos(x)+\sin(x)\cos(x)}{2}$$

We have to show with the conditions : $0<x<\pi$ and $0<y<\pi$ and $0<z<\pi$ and $\tan(\frac{x}{2})\tan(\frac{y}{2})\tan(\frac{z}{2})=1$

:

$$\sum_{cyc}\frac{\sin(x)+\cos(x)+\sin(x)\cos(x)}{2}\leq 1.5$$

But since $$\sum_{cyc}\frac{\sin(x)+\cos(x)}{2}\leq 1.5$$

Because $h(x)=\sin(x)+\cos(x)$ is concave on $[0,0.75\pi]$ we have:

$$\sum_{cyc}\frac{\sin(x)+\cos(x)}{2}\leq 3\frac{\sin(\frac{x+y+z}{3})+\cos(\frac{x+y+z}{3})}{2}$$

And the same reasoning as below conducts to

$$\sum_{cyc}\frac{\sin(x)+\cos(x)}{2}\leq 3\frac{\sin(\frac{x+y+z}{3})+\cos(\frac{x+y+z}{3})}{2}\leq 1.5$$

Remains to show that :

$$(\sin(2x)+\sin(2y)+\sin(2z))0.25\leq 0$$

Can you end now ?