Stability without Lyapunov methods
I assume that your change of coordinates is well defined. I did not check it but it is something you should really do when using change of coordinates. Can you write here the transformation you used?
Remeber the definition of stability of a point p:
$\forall \epsilon>0 , \exists \delta>0$ s.t. $\forall x_0\in B_{\delta}(p), \phi(t, x_o) \in B_{\epsilon}(p)$ for all time $t>0$.
Down here I will write symbolically the idea that no matter how close to your equilibrium you start, you can choose an $\epsilon$ ball that does not contain the flow (you can think of this as a definition of instability):
$\exists \epsilon>0 , \forall \delta>0$ $\exists x_0\in B_{\delta}(p), \phi(t, x_o) \notin B_{\epsilon}(p)$ for some time $t>0$.
Now, fix $\epsilon=1/2$, $p=(1,0)$. Remember that if you choose a point on the circumference with radius $1$, the flow stays on the circumference since $\dot{r}=0$, therefore we are only interested in the $\theta$ dynamics.
Consider a generic ball $B_{\delta}(p)$. No matter how you choose this ball, you can always find a point $x_0=(1, \theta_0) \in B_{\delta}(p)$. (I suggest you to find an explicit way of finding such an $x_0$).
Therefore $\phi(t, x_0)$ will have the form $(1, \theta(t))$. The final step consists in showing that such a flow 'goes out' of the $\epsilon$-ball! For example, we can show that in finite time the point will reach the point $(1, \pi/2 )$ that is clearly out of our $\epsilon=1/2$-ball.
How do we show this?
- $\theta(t)$ is monotone non-decreasing since $sin^2(θ/2)\geq0$
- $\dot{\theta}(t)\geq sin^2(θ_0/2)\ \forall \theta \in [\theta_0, \pi/2]$
Since $\theta(t)$ is monotone non decreasing, it will rather (a) go to infinity or (b) have a finite limit $\tilde{\theta} $. In case (a) you are done, you will definitily go out of your $\epsilon$-ball.
Case (b) requires a small argument. We claim here that $\tilde{\theta} > \pi/2$.
Assume (by contraditction) $\tilde{\theta}\leq\pi/2$. Then $\theta(t) \in [\theta_0, \tilde{\theta}]\ \forall t>0$.
Recalling 2. :
$\dot{\theta}(t)\geq sin^2(θ_0/2) \implies \theta(t) \geq \theta_0 + sin^2(\theta_0)t$
The last inequality it is clearly a contradiction: $\theta$ grows unbounded, against the initial claim that $\theta(t)$ had a finite limit.
Remark: In the last proof by contradiction we did not prove that there is no finite limit for $\theta(t)$, but rather that this limit can not be smaller than $\pi/2$. Indeed, one can prove that $\theta$ converges to $2\pi$. As an exercise you could try to prove this fact with a similar contradiction argument. It is a common argument that is used (for example) to show asymptotic stability of an equilibrium using Lyapunov Functions!
For the proof of stability, we can actually show that given any point in any delta ball around $p$, the solution will converge to (1,0). Indeed, we can prove that $\theta$ goes to $2\pi$ by simply adapting the argument above. For the convergence of radius, consider that for $r<1$ $r(t)$ is monotone increasing and bounded from above. The limit is clearly $r=1$. Analogous result is for $r>1$. Actually the function is odd, so the flow is simmetric.