For acute $\triangle ABC$, prove $(\cos A+\cos B)^2+(\cos A+\cos C)^2+(\cos B+\cos C)^2\leq3$

Let $a^2+b^2-c^2=z$, $a^2+c^2-b^2=y$ and $b^2+c^2-a^2=x$.

Thus, we need to prove that $$\sum_{cyc}\left(\frac{b^2+c^2-a^2}{2bc}+\frac{a^2+c^2-b^2}{2ac}\right)^2\leq3$$ or $$\sum_{cyc}\left(\frac{x}{2\cdot\sqrt{\frac{x+z}{2}}\cdot\sqrt{\frac{x+y}{2}}}+\frac{y}{2\cdot\sqrt{\frac{y+z}{2}}\cdot\sqrt{\frac{x+y}{2}}}\right)^2\leq3$$ or $$\sum_{cyc}\left(\frac{x}{\sqrt{(x+y)(x+z)}}+\frac{y}{\sqrt{(x+y)(y+z)}}\right)^2\leq3$$ or $$\sum_{cyc}(x\sqrt{y+z}+y\sqrt{x+z})^2\leq3\prod_{cyc}(x+y)$$ or $$\sum_{cyc}\left(2x^2y+2x^2z+2xy\sqrt{(x+z)(y+z)}\right)\leq\sum_{cyc}(3x^2y+3x^2z+2xyz)$$ or $$2\sum_{cyc}xy\sqrt{(x+z)(y+z)}\leq\sum_{cyc}(x^2y+x^2z+2xyz),$$ which is true by AM-GM: $$2\sum_{cyc}xy\sqrt{(x+z)(y+z)}\leq\sum_{cyc}xy(x+z+y+z)=\sum_{cyc}(x^2y+x^2z+2xyz).$$

Welcome to MSE!

A very brute-force approach (basically Lagrange Multipliers): put $C=\pi-A-B$. The condition that the triangle is acute imposes $0\leq A,B\leq \pi/2$ and $A+B\geq \pi/2$ (except we can't have both $A=B=\pi/2$), so you are trying to maximize the function in this triangle. (Note: we are expecting the 'symmetric' solution $A=B=C=\pi/3$, or else an isosceles right triangle.) An ugly computation shows the gradient has four critical points in this region, the maximum of which occurs at $(\pi/3,\pi/3)$. The boundary sides of the triangle are very nice as well: the function reduces to $2+\sin(2A)$, or $2+\sin(2B)$ by symmetry. So the maximum solution is the symmetric one.