Prove $\sqrt{a^2+4}+\sqrt{b^2+4}\leq\frac{\sqrt{2}}{4}(a+b)^2$ with $\frac1a+\frac1b=1$.

Let $x=\frac{1}{a}$ and $y=\frac{1}{b}$. We want to show that $x+y=1$ and $x,y\geq 0$ imply

$$ y^2x\sqrt{4x^2+1}+x^2y\sqrt{4y^2+1}\leq \frac{\sqrt{2}}{4}. $$

Letting $x=\frac{1+t}{2},y=\frac{1-t}{2}$, this is equivalent to finding the maximum of $$ f(t) =(1-t^2)\left[ (1-t)\sqrt{1+(1+t)^2}+(1+t)\sqrt{1+(1-t)^2}\right]$$ (which is an even function) over $[-1,1]$. We have $$ f(t) = (1-t^2)\sqrt{4-2t^2+2t^4+2(1-t^2)\sqrt{1+t^4}} $$ $$ f(\sqrt{t}) = \sqrt{2}(1-t)\sqrt{2-t+t^2+(1-t)\sqrt{1+t^2}}$$ and both $1-t$ and $2-t+t^2+(1-t)\sqrt{1+t^2}$ are positive and decreasing functions over $[0,1]$, so the maximum of $f(t)$ is attained at the origin. Indeed $$2-\tan\theta+\tan^2\theta+(1-\tan\theta)\sqrt{1+\tan^2\theta} = \frac{2+\sqrt{2}\sin\left(\tfrac{\pi}{4}+\theta\right)}{2\sin^2\left(\frac{\pi}{4}+\frac{\theta}{2}\right)}$$ and the derivative of the RHS is $$ -\frac{1}{\sin^2\left(\frac{\pi}{4}+\frac{\theta}{2}\right)\left(1+\tan\frac{\theta}{2}\right)}<0.$$

By AM-GM and C-S we obtain: $$\frac{\sqrt2(a+b)^2}{4}=\frac{\sqrt{2(a+b)}\sqrt{a+b}(a+b)^2}{4ab}\geq(\sqrt{a}+\sqrt{b})\sqrt{a+b}=$$ $$=\sqrt{a^2+ab}+\sqrt{b^2+ab}\geq\sqrt{a^2+\frac{4a^2b^2}{(a+b)^2}}+\sqrt{a^2+\frac{4a^2b^2}{(a+b)^2}}=$$ $$=\sqrt{a^2+4}+\sqrt{b^2+4}.$$

To get rid of some square roots and simplify, substitute $(a,b)\to (2x, 2y)$. The inequality can be written as:

Let $x,y >0$ and $\frac{1}{x}+\frac{1}{y} = 2$. Prove that:

$$\sqrt{2(x^2+1)}+\sqrt{2(y^2+1)} \leq (x+y)^2$$

In this case, from AM-GM we can see that:

$$\sqrt{2(x^2+1)}+\sqrt{2(y^2+1)} \leq \frac{x^2+1+2}{2}+\frac{y^2+1+2}{2} = \frac{x^2+y^2}{2}+3$$

It will be enough to prove:

$$\frac{x^2+y^2}{2}+3 \leq (x+y)^2$$

or

$$x^2+y^2+4xy \geq 6$$

From AM-GM on the initial condition, we can see that $xy \geq 1$. Therefore

$$x^2+y^2+4xy \geq 6xy \geq 6$$

Equality occurs when $(x,y) = (1,1)$, so when $(a,b)=(2,2)$.