Is there a unique solution for this quadratic matrix equation?

You could note that the matrix similarity \begin{align} \pmatrix{I & \mathbf{0} \\ X & -I}\pmatrix{\mathbf{0} & I \\ -C & -B} & \pmatrix{I & \mathbf{0} \\ X & -I} & \\ =\pmatrix{\mathbf{0} & I \\ C & X + B} & \pmatrix{I & \mathbf{0} \\ X & -I} & \\ =\pmatrix{ X & -I \\ X^2 + BX + C& -X - B} & \\ \end{align}

gives your equation in $X$, and if the equation is solved, then the matrix $\pmatrix{0 & I \\ -C & -B}$ is block diagonalized. Also note that there is a closed form solution to bring (almost) any matrix into the form $\pmatrix{0 & I \\ -C & -B}$ through a similarity transform. I do not know what this form is called in the literature, but I like to call it the block companion form. Here is how to do it

\begin{align} \pmatrix{G^{-1} & \mathbf{0} \\ G^{-1}M & I}\pmatrix{M & G \\ F & D} \pmatrix{G & \mathbf{0} \\ -G^{-1}MG & I} & \\ =\pmatrix{G^{-1}M & I \\ G^{-1}M^2 + F & G^{-1}MG + D} \pmatrix{G & \mathbf{0} \\ -G^{-1}MG & I} & \\ =\pmatrix{ \mathbf{0} & I \\ G^{-1}M^2G + FG - G^{-1}M^2G - DG^{-1}MG & D+G^{-1}MG} & \\ =\pmatrix{ \mathbf{0} & I \\ (F-DG^{-1}M)G & D+G^{-1}MG} & \\ =\pmatrix{0 & I \\ -C & -B} & \\ \end{align}

If a solution to $X^2 + BX + C = \mathbf{0}$ were possible in a closed form here, then it could split the eigenvalue problem in half, be applied recursively and thus have a closed form solution to the eigenvalue problem. The existence of such a solution has already been dis-proven by the Abel-Ruffini Theorem.

Since diagonalization is a difficult problem without a closed form solution, it is no wonder you had such difficulty finding such a solution. Maybe this is why the equation is related to what is named the quadratic eigenvalue problem as J.M. noted.

An answer by year!

Consider the Riccati equation (1) $XAX+XB+CX+D=0_n$ where $X\in M_n(\mathbb{C})$ is unknown and $A,B,C,D\in M_n(\mathbb{C})$ are generic given matrices. (that is, the entries of these matrices are independent commutative indeterminates OR, more simply, there are no algebraic relations with coefficients in $\mathbb{Q}$ linking the entries of $A,B,C,D$).

Then (1) can be reduced to an equation in the form (2) $X^2+AX+B=0_n$ where $A,B$ are generic matrices. This equation has exactly $\binom{2n}{n}$ solutions and solving (2) can be reduced to solve a polynomial equation $P(u)=0$ with $degree(P)=2n$ and Galois group $S_{2n}$.

Note that , in your equation (3) $AX^2+BX+C=0_n$, if $A$ is generic, then $A$ is invertible and (3) has in fact the form (2).