Find the Minimum Value of $x^2+y^2$
I give here a geometrical solution using the auxiliary plane with coordinates $(X;Y)$ where
$$X=x+y \ \ \text{and} \ \ Y=xy$$
(see figure below).
Important : In this plane, due to arithmetic-geometric means inequality, only region defined by $Y\leq \tfrac14X^2$ corresponds to real solutions. We will come back to this restriction later on.
Using variables $X,Y$, the issue can be re-formulated in this way :
Minimize
$$m:=x^2+y^2=X^2-2Y \tag{1}$$
knowing that
$$\begin{pmatrix}X\\Y\end{pmatrix}=2\begin{pmatrix}\sin(a)\\ \cos(a)\end{pmatrix}+\begin{pmatrix}-\cos(b)\\ \ \ \ \sin(b)\end{pmatrix}\tag{2}$$
The RHS of (2) can be given a geometric interpretation as the combination of two circular movements : a planet with orbital radius $2$ and its satellite with orbital radius $1$, yielding an "allowed" annular region $(A)$ with internal/external radii $1$ and $3$ resp. (or in an equivalent way as the zone swept by the tool of a two-arms robot with the same characteristics).
Condition (1) can be rewritten under the form :
$$\text{find the minimal} \ m \ \text{such that} \ \ Y=\tfrac12X^2-\tfrac12 m \ \text{intersects region (A)}\tag{3}$$
The equation in (3) can be interpreted as the generic equation of a family of parabolas $P_m$, "sweeping" the plane with a vertical move. Our problem boils down to find the "ultimate" parabola intersecting the annular region.
It is evident on the figure that this parabola $P_m$ corresponds to $m=-6$ (point of tangency $Q$) (I don't do the corresponding calculations). The values of $a$ and $b$ can be deduced without difficulty from this value of $m$.
BUT, in the considered case, we haven't taken into account whether solutions in $x,y$ are real or complex. In the case $m=-6$, we have complex solutions $x,y$ as remarked by @Christian Blatter.
A reasonable objective would be to have real solutions for $x,y$. In this case the minimal value of $m$ is quite different.
We will find it rather easily by restricting investigations to the intersection of the annular region $(A)$ with the region below parabola with equation $Y=X^2/4$, that we call the feasible region (inside green border crescent).
In this case, the values of expression (1) are visibly situated between two extremal values that can be computed as being $m_1=-4+2\sqrt{5}$ and $m_2=10$ corresponding to parabolas $P_m$ passing through points $P$ and $R$ resp.
Thus, the minimal value of $m=x^2+y^2$ compatible with real solutions to the given system is $m_1=-4+2\sqrt{5} \approx 0.4721$.
Remarks :
1) Extremal value $m_2=10$ is visible on your graphical representation (reminding egg-boxes...).
2) The (sum;product) representation plane we have used is of considerable importance when using $trace = \lambda_1+\lambda_2$ and determinant $\lambda_1\lambda_2$ of a $2 \times 2$ matrix, especially for a taxinomy of linear systems. It is called the "trace-determinant space". See among many others p. 74 of this interesting PhD thesis.