converting a differential equation to polar coordinates

The easiest way to do the polar form change is to differentiate $r^2 = x^2+y^2$ and hence $r' = (xx' + yy')/r$. When you substitute for $x,y$ you should find $$ r'= r(1-r) + \epsilon r^2\sin\theta.$$ When $\epsilon = 0$ the dynamics of $r$ decouples from $\theta$ and we can see we have a unstable fixed point(in $r$) at $r=0$ and a stable (and hence attracting) fixed point at $r=1$.

We can do the same thing by differentiating $\tan\theta = y/x$ and hence $r^2\theta' = y'x-x'y $. Again, substituting gives $$\theta' = 1 + \epsilon(1 + r\cos\theta).$$ When $\epsilon = 0$, $\theta' =1$ and $\theta = t+ \theta_0$.

Now you can use the face that $r=1$ implies $r'=0$, and use the initial conditions for $\theta_0$ to answer the question.

$$\left ( \begin{array}\\ \cos{\theta} & -r \sin{\theta} \\ \sin{\theta} & r \cos{\theta} \end{array} \right )^{-1}= \left ( \begin{array}\\ \cos{\theta} & \sin{\theta} \\ \frac{-1}{r}\sin{\theta} & \frac{1}{r} \cos{\theta} \end{array} \right )$$

Multiplied on the left side, simplified and obtained: $$\left ( \begin{array}\\ \cos{\theta} & \sin{\theta} \\ \frac{-1}{r}\sin{\theta} & \frac{1}{r} \cos{\theta} \end{array} \right ) * \left ( \begin{array}\\ r\cos\theta(1-r)-r\sin\theta-\epsilon r \sin\theta\\r\sin\theta(1-r) + r\cos\theta + \epsilon(r\cos\theta+r^2) \end{array} \right )$$ $$\left ( \begin{array} *r\cos^2\theta (1-r)-r\sin\theta\cos\theta-\epsilon r \sin\theta\cos\theta+r\sin^2\theta(1-r) + r\cos\theta\sin\theta + \epsilon\sin\theta (r\cos\theta+r^2) \\ -\sin\theta \cos\theta(1-r)+\sin^2\theta+\epsilon \sin^2 \theta+\sin\theta\cos\theta(1-r)+\cos^2\theta+\epsilon(\cos^2 \theta+r\cos\theta) \end{array} \right )$$

Cancel out: $$\left ( \begin{array} +r(1-r) + \epsilon\sin\theta r^2 \\ +1+\epsilon+\epsilon r\cos\theta \end{array} \right )$$ Doesn't equal: $$\left ( \begin{array}\\ r' \\ \theta' \end{array} \right )=\left ( \begin{array} \\ r(1+\epsilon) \\1 + \epsilon r \end{array} \right )$$