Finding conditions to make roots of a quadratic less than one in magnitude

A quick thought: Let $y=(x-α)(x-β)$

Our roots are $x=α$ and $x=β$, and we want their magnitudes to be less than 1.

We also have $b=α+β$ and $c=αβ$. If $|\,b\,|≥2$, we have $2≤|\,b\,|=|\,α+β\,|≤|\,α\,|+|\,β\,|$ so either $|\,α\,|≥1$ or $|\,β\,|≥1$.

For the condition on $c$, one comes from the discriminant as you mentioned, and you can get the other by looking at the graph of $y$ at $x=1$ and $x=-1$.

There is a generic way to solve this classical problem:

Find conditions on the coefficients of a quadratic polynomial $p(x)=ax^2+ bx+c$ so that it has two real roots between $x_0$ and $x_1$.

1. A first condtiion is, of course, that it has two real roots: its discriminant $\Delta$ must be positive.
2. Let $\alpha<\beta\,$ be the real roots. A second condition is that $x_0$ and $x_1$ must be outside the interval $[x_0, x_1]$. This means: $$ap(x_0)>0, \quad ap(x_1)>0 .$$
3. The last condition is that $\,x_0<\alpha$ and $\beta>x_1$. Knowing 2, this is equivalent to $$x_0<\dfrac{\alpha+\beta}2<x_1\iff x_0<-\frac b{2a}<x_1$$

In the present case, all this translates to: $$ \begin{cases} b^2>4c\\ b+c>-1,\quad b-c<1\\-2<b<2 \end{cases} $$

Graphical representation of the solutions as a domain of the plane: