Solving a quadratic matrix equation
This relationship is called congruence, and for symmetric (or hermitian) matrices $A, B$ such an $X$ exists if and only if the matrices have the same inertia (the same number of positive, negative, zero eigenvalues). Fro general matrices congruence is much less well known, but it is quite well understood, see:
http://gauss.uc3m.es/web/personal_web/fteran/papers/2010-1.pdf
I read the reference of Igor, but I have nothing shot about our equation !
There is a case that seems less difficult. If $A,B$ are symmetric complex, then the problem is essentially equivalent to solve $2$ equations of the form $ZZ^T=D$, that is, the case $A=I$ or $B=I$.
equation 1: $YY^T=A$ , equation 2: $ZZ^T=B$ , equation 3: $XY=Z$. If $A,B$ are invertible, then $Y,Z$ exist and $X=ZY^{-1}$.In these conditions, can we find, at least in theory, all solutions ?
For Halder. The equation $XAX=B$ is easy to solve (at least when $A$ is invertible) because it can be rewritten $(XA)^2=BA$.
EDIT: Let $A,B\in\mathcal{M}_n(\mathbb{C})$. According to a result by Turnbull, Aitken (cited in the Igor's reference) our equation admits a solution iff $A,B$ are congruent iff there are invertible $P,Q$ s.t. $PAQ=B,PA^TQ=B^T$. Then, for generic $A,B$, the equation $XAX^T=B$ has no solutions. Thus, the "good" equation is $XAX^T=CAC^T$. Using my computer, I randomly choose real matrices $A,C$ ; I find that the algebraic set of complex solutions has dimension $1$ when $n=2$, $1$ when $n=3$, $2$ when $n=4$, $2$ when $n=5$.