Solving a quadratic 9-equation system
Better than the QR decomposition I suggested above, the SVD decomposition might work. In fact, given
$$
{\bf A} = {\bf U}\,{\bf \Sigma }\;\overline {\bf V}
$$
the system above becomes
$$
\left\{ \matrix{
\overline {\bf A} \;{\bf A} = {\bf V}\,{\bf \Sigma }\;\overline {\bf U} {\bf U}\,{\bf \Sigma }\;\overline {\bf V} = {\bf V}\,{\bf \Sigma }^{\bf 2} \,\overline {\bf V} = {\bf B}\quad \Leftrightarrow \quad {\bf B}\;{\rm diagonalizable} \hfill \cr
{\bf U}\,{\bf \Sigma }\;\overline {\bf V} \,{\bf x} = {\bf y}\quad \Leftrightarrow \quad \overline {\bf x} \;{\bf V}\,{\bf \Sigma }^{\bf 2} \,\overline {\bf V} \,{\bf x} = \overline {\bf x} \;{\bf B}\,{\bf x} = \overline {\bf y} \;{\bf y} \hfill \cr} \right.
$$
so that B,x,y are not totally independent (for real solutions to exist).
Therefrom ${\bf \Sigma }$ and ${\bf V}$ can be obtained from the normalized eigenvalue decomposition of ${\bf B}$.
${\bf U}$ is the rotation matrix that brings ${\bf \Sigma }\;\overline {\bf V} \,{\bf x}$ in ${\bf y}$.