Find the minimum value of $a^2+b^2$

Edit : I should have added a condition that $|-a/2|\ge 2$.

---

I think that your idea is nice, but note that $(2)$ is incorrect. Using $x+\frac 1x=y$, we have $$y^2+ay+b\color{red}{-2}=0\tag3$$

So, we have to have $$a^2-4(b-2)\ge 0\iff b\le \frac{a^2}{4}+2\tag4$$

We want $(3)$ to have at least one real solution $y$ such that $|y|\ge 2$ as you wrote.

Let $f(y):=y^2+ay+b-2$. The condition is represented as $$\left|-\frac a2\right|\ge 2\quad\text{or}\quad f(-2)\le 0\quad\text{or}\quad f(2)\le 0$$$$\iff |a|\ge 4\quad\text{or}\quad 2-2a+b\le 0\quad\text{or}\quad 2+2a+b\le 0\tag5$$

Hence, drawing $(4)(5)$ in $ab$-plane gives that the minimum value of $a^2+b^2$ is $$\left(\frac{|2\pm 2\cdot 0+0|}{\sqrt{(\pm 2)^2+1^2}}\right)^2=\color{red}{\frac 45}$$ (note here that $a^2+b^2$ represents the square of the distance from $(0,0)$ to $(a,b)$)

for $(a,b)$ such that $a^2+b^2=\frac 45$ and $b=\pm 2a-2$, i.e. $(a,b)=(\pm 4/5,-2/5)$.

Divide the equation by $x^2$ $$ \begin{align} 0 &=x^2+ax+b+\frac ax+\frac1{x^2}\\ &=\left(x+\frac1x\right)^2+a\left(x+\frac1x\right)+(b-2)\tag{1} \end{align} $$ Since $\left|\,x+\frac1x\,\right|\ge2$ and the solutions to $(1)$ are $$ x+\frac1x=\frac{-a\pm\sqrt{a^2-4b+8}}2\tag{2} $$ and since the sign of $a$ does not change $a^2+b^2$, and only changes the sign of $x$, we can assume that $a\ge0$. Then we need $$ a+\sqrt{a^2-4b+8}\ge4\tag{3} $$ Since $a=1$ and $b=0$ satisfy $(3)$, the minimum of $a^2+b^2$ is at most $1$. Therefore, we can assume that $a\le1$ and $b\ge-1$. Then $(3)$ becomes $$ a^2-4b+8\ge16-8a+a^2\iff a\ge\frac{b+2}2\tag{4} $$ Therefore, $$ a^2+b^2\ge\left(\frac{b+2}2\right)^2+b^2\ge\frac45\tag{5} $$ Since $a=\frac45$ and $b=-\frac25$ satisfies $(3)$ and $a^2+b^2=\frac45$, we get $$ \min\!\left(a^2+b^2\right)=\frac45\tag{6} $$