Poincaré-Bendixon show periodic solutions.

What you've done so far is correct, but your conclusions are not quite right. You found that

$$ \dot{r} = r - \frac{r^3}{2} ( 1 + \cos^2(2 \theta) ) $$

When you found that " $r_{min} = 1$ ", what you should do is show $\dot{r} > 0$ when $r < 1$. When you found that " $r_{max} = \sqrt{2}$ ", what you should truly do is show that $\dot{r} < 0$ when $r > \sqrt{2}$.

Then, if you have a system whose initial radius lies in the interval $(1,\sqrt{2})$, then the radius of any point on its trajectory must lie in the interval $[1, \sqrt{2}]$. Thus the trajectory is "trapped" in this region. Now we can apply the Poincare-Bendixson theorem. The point here is that even when $\dot{r} = 0$, this doesn't mean we have a fixed point, unless the angle $\theta$ is also unchanging. Indeed, you noted that the origin is the only fixed point, so by the Poincare-Bendixson theorem, we have a bounded trajectory with no fixed points; there must be a limit cycle.

Dynamical system

Find the periodic solution for the dynamical system: $$ % \begin{align} % \dot{x} &= x - y - x^{3} \\ % \dot{y} &= x + y - y^{3} \\ % \end{align} \tag{1} % $$

Fixed points

There is a lone fixed point at the origin $$ \left[ \begin{array}{c} \dot{x} \\ \dot{y} \end{array} \right]_{(0,0)} = \left[ \begin{array}{c} 0 \\ 0 \end{array} \right] $$

Methods

Compute $\dot{r}$.

Use the polar coordinate transform $$ % \begin{align} % x &= r \cos \theta \\ % y &= r \sin \theta \\ % \end{align} % $$ which implies $$ r^{2} = x^{2} + y^{2} \tag{2} $$ Compute the derivative with respect to time for $(2)$ and use the definitions in $(1)$. This leads to $$ \dot{r} = r - r^{3} \left( \cos^{4} \theta + \sin^{4} \theta \right) = r \left( 1 - \frac{1}{4} \left( 3 + \cos 4 \theta \right) r^{2} \right) \tag{3} $$

Trapping region

To categorize the flow, identify regions where $r$ is expanding or shrinking. Consider the limiting values $$1 \le \cos 4\theta \le 1.$$

1. $\cos 4\theta = 1$

Equation $(3)$ reduces to $$ \dot{r} = r \left( 1 - r^{2} \right) $$ The conclusion is $$ \begin{cases} \color{blue}{\dot{r} > 0} & \color{blue}{r < 1} \\ \color{red}{\dot{r} < 0} & \color{red}{r > 1} \\ \end{cases} $$

2. $\cos 4\theta = -1$

Equation $(3)$ reduces to $$ \dot{r} = r \left( 1 - \frac{1}{2}r^{2} \right) $$ This implies $$ \begin{cases} \color{blue}{\dot{r} > 0} & \color{blue}{r < \sqrt{2}} \\ \color{red}{\dot{r} < 0} & \color{red}{r > \sqrt{2}} \\ \end{cases} $$

The figures below depict the zone structure for the two limiting conditions. Combine these zone structures and identify where, in both figures, is $\dot{r}$ always positive, and always negative. There will be an annulus where the sign of $\dot{r}$ isn't specified. The third figure shows this trapping region in gray.

The trapping region is the set $ 1 \le r \le \sqrt{2}$

Results

The flow field is plotted next. The dashed, red lines are the nullclines. The gray annulus is the trapping region.