The polynomial $x^3 + 2ax^2 + (2a^2 + b)x + c$ has three (real and not necessairily distinct) roots and $b$ is one of them. Prove that $(ac)^2 \le 3$.

That $b$ is a root implies $-c = b^3+2ab^2+b^2+2a^2b$. Then $x^3+2ax^2+(2a^2+b)x+c = (x-b)(x^2+(2a+b)x+(2a^2+b+2ab+b^2)$ is valid. Since both roots of the quadratic are real, the discriminant must be non-negative: $(2a+b)^2-4(2a^2+b+2ab+b^2) \ge 0$, which is equivalent to $a \in [\frac{-b-\sqrt{-2b^2-4b}}{2},\frac{-b+\sqrt{-2b^2-4b}}{2}]$. We therefore must have $b \le 0$. $\bf\text{For ease, replace $b$ by $-b$}$.

Our problem comes down to maximizing $$(ab^3-2a^2b^2-ab^2+2a^3b)^2$$ given that $$a \in [\frac{b-\sqrt{4b-2b^2}}{2},\frac{b+\sqrt{4b-2b^2}}{2}].$$ Note that we in particular must have $b \in [0,2]$. Let $f(a) = ab^3-2a^2b^2-ab^2+2a^3b$. Then $f'(a) \ge 0$ if and only if $6a^2-4ba+b^2-b \ge 0$, which has roots $\frac{2b\pm\sqrt{6b-2b^2}}{6}$. Now, for any $b \in [0,2]$, $\frac{2b-\sqrt{6b-2b^2}}{6} < \frac{2b+\sqrt{6b-2b^2}}{6} < \frac{b+\sqrt{4b-2b^2}}{2}$, and it holds that $\frac{b-\sqrt{4b-2b^2}}{2} < \frac{2b-\sqrt{6b-2b^2}}{6}$ if and only if $ b \in [0,\frac{50}{33})$. Since we want to maximize $f(a)^2$, it suffices to check extremal $a$ and $a$ for which $f'(a) = 0$, since $(f^2)' = 2ff'$ and $f(a)=0$ implies $(ac)^2 = 0 \le 3$. That is, we just have to show $f(a)^2 \le \sqrt{3}$ for $a = \frac{b+\sqrt{4b-2b^2}}{2},\frac{2b+\sqrt{6b-2b^2}}{6},\frac{2b-\sqrt{6b-2b^2}}{6}$, and for $a = \frac{b-\sqrt{4b-2b^2}}{2}$ when $b \in (\frac{50}{33},2]$.

And this is easily doable. For example, at $a = \frac{b+\sqrt{4b-2b^2}}{2}$, $f(a) = \frac{1}{4}(2-b)b^2(b+\sqrt{2b(2-b)})$. Since $(2-b)b \le 1$, $f(a) \le \frac{1}{4}b(b+\sqrt{2})$, which is $\le \sqrt{3}$. The other values of $a$ can also be handled easily.


$\color{brown}{\textbf{Formulation of the problem.}}$

Let $$P(x,a,b,c) = x^3+2ax^2+(2a^2+b)x+c.$$

By the condition, $P(b,a,b,c) =0,$ $$b^3+(2a+1)b^2+2a^2b+c=0.\tag1$$

Also, in accordance with the Besou theorem, $$P(x,a,b,c) = (x-b)(x^2+(b+2a)x+b^2+2ab+2a^2+b).$$

By the condition, $P(x,a,b,c)$ has three (real and not necessairily distinct) roots, so the discriminant of the quadratic factor is non-negative, $$(b+2a)^2 - 4(b^2+2ab+2a^2+b)\ge 0,$$ $$(b+2a)^2 - 2(b+2a)^2 - 2b^2 - 4b \ge 0,$$ $$(b+2a)^2+2b^2+4b \le 0.\tag2$$

On the other hand, from $(1)$ should $$c=-\dfrac b2((2a+b)^2+b^2+2b),$$ $$ac = f(a,b) = -\dfrac12ab\left((2a+b)^2+b^2+2b\right).\tag3$$

$\color{brown}{\textbf{Inner extremums.}}$

The stationary points of $f(a,b)$ can be defined from the system \begin{cases} b\left((2a+b)^2+b^2+2b\right)+4ab(2a+b) = 0\\ a\left((2a+b)^2+b^2+2b\right)+2ab(2a+b+b+1) = 0. \end{cases}

The greatest value of $f^2(a,b)$ can not be achieved for $(a=0)\vee (b=0),$ so \begin{cases} (2a+b)^2+b^2+2b+4a(2a+b) = 0\\ (2a+b)^2+b^2+2b+2b(2a+2b+1) = 0, \end{cases}

$$\begin{cases} 12a^2+8ab+2b^2+2b= 0\\ 4a^2=2b^2+b \end{cases}\Rightarrow \begin{cases} 8ab+8b^2+5b= 0\\ 4a^2=2b^2+b \end{cases}\Rightarrow \begin{cases} a = -b-\dfrac58\\ 64b^2+80b+25= 32b^2+16b, \end{cases} $$ $$32b^2+64b+25=0,\quad 32(b+1)^2=7,$$ $$\begin{pmatrix}a\\b\end{pmatrix} \in \left\{ \begin{pmatrix}\dfrac{-13-\sqrt{14}}8\\ \dfrac{-8-\sqrt{14}}8\end{pmatrix} \begin{pmatrix}\dfrac{-13+\sqrt{14}}8\\ \dfrac{-8+\sqrt{14}}8\end{pmatrix} \right\}.\tag4$$

Substitution of the stationary points $(4)$ in $(2)$ shows that they are not in the area.

$\color{brown}{\textbf{Extremums in the bound.}}$

Extremums of $f(a,b)$ on the area bound can be defined by the Lagrange multipliers method, as the stationary points of the function $$g(a,b,\lambda) = \dfrac a2(b^3+2b^2) + \lambda\left((b+2a)^2+2b^2+4b\right),\tag5$$

or from the system \begin{cases} \dfrac b4(2b^2+4b)+\lambda(4b+8a) = 0\\ \dfrac a2(3b^2+4b) + \lambda(2b+4a+4b+4) =0 \\ (b+2a)^2+2b^2+4b = 0. \end{cases}

Unknown $\lambda$ can be eliminated: \begin{cases} -b(b+2a)^2+16\lambda(b+2a) = 0\\ ab(3b+4) + 4\lambda(3b+2a+2) =0 \\ (b+2a)^2+2b^2+4b = 0, \end{cases}

\begin{cases} 16\lambda = b(b+2a)\\ 4a(3b+4) + (b+2a)(3b+2a+2) =0 \\ (b+2a)^2+2b^2+4b = 0, \end{cases}

\begin{cases} 4a(3b+4) - 2b^2-4b + (b+2a)(2b+2) =0 \\ (b+2a)^2+2b^2+4b = 0, \end{cases}

\begin{cases} 16ab+20a -2b =0\\ (b+2a)^2+2b^2+4b = 0. \end{cases}

Therefore, \begin{cases} a=\dfrac{b}{8b+10}\\ b\left(1+\dfrac1{4b+5}\right)^2+2b+4 = 0\\ b\in\left[-2,0\right].\tag6 \end{cases}

This leads to the cubic equation for $b:$ $$2b(2b+3)^2+(b+2)(4b+5)^2 = 0,\quad b\in\left[-2,0\right],$$ $$24b^3+96b^2+123b + 50 = 0,\quad b\in\left[-2,0\right],$$

which can be presented in the trigonometric form: $$8(3b+4)^3-15(3b+4) = 2,\quad 3b+4\in\left[-2,4\right],$$ $$4\left(\sqrt{\dfrac25}(3b+4)\right)^3-3\left(\sqrt{\dfrac25}(3b+4)\right) = \dfrac25\sqrt{\dfrac25}, \quad \sqrt{\dfrac25}(3b+4)\in\left[-2\sqrt{\dfrac25},4\sqrt{\dfrac25}\right],$$ $$\cos\left(3\arccos\left(\sqrt{\dfrac25}(3b+4)\right)\right) = \dfrac25\sqrt{\dfrac25}, \quad \sqrt{\dfrac25}(3b+4)\in\left[-2\sqrt{\dfrac25},4\sqrt{\dfrac25}\right].$$

Obtained system has the solutions $$\sqrt{\dfrac25}(3b+4) \in \cos \left\{\begin{matrix} \dfrac13\left(2\pi - \arccos \dfrac25\sqrt{\dfrac25}\right),\\ \dfrac13\arccos \dfrac25\sqrt{\dfrac25},\\ -\dfrac13\arccos \dfrac25\sqrt{\dfrac25} \end{matrix}\right\},$$

or $$b \in \dfrac13\sqrt{\dfrac52}\cos \left\{\begin{matrix} \dfrac13\left(2\pi - \arccos \dfrac25\sqrt{\dfrac25}\right),\\ \dfrac13\arccos \dfrac25\sqrt{\dfrac25},\\ -\dfrac13\arccos \dfrac25\sqrt{\dfrac25} \end{matrix}\right\}-\dfrac43.\tag 7$$

Substitution of the stationary points $(7)$ to $(6)$ and $(5)$ gives (approximately) $$\begin{pmatrix}a\\b\\g^2(a,b,0)\end{pmatrix} =\left\{ \begin{pmatrix}1.34369\\-1.37821\\0.629631\end{pmatrix}, \begin{pmatrix}-0.271691\\-0.856116\\0.0129713\end{pmatrix}, \begin{pmatrix}-0.271691\\-0.856116\\0.0129713\end{pmatrix} \right\}.\tag8$$

Therefore, in the given conditions $$\color{brown}{\mathbf{(ac)^2 < 0.63}}.$$