Largest circle in basin of attraction of the origin.
(This is only a "numerical" answer to the question)
A basin of attraction can be found numerically using reverse-time (or backward) integration. If we choose some initial point and integrate backwards for a sufficient time, we can obtain the set of the system states that it had before it reached the selected initial point. So, if we choose sufficiently many initial points that are contained in the basin of attraction (i.e. sufficiently close to the attracting steady state) and integrate backwards, we can get some idea of how the basin of attraction looks like.
The following picture demonstrates the aprroximation of the basin of attraction of the system: The blue curves fill the basin of attraction. The red circle is the largest circle that fits in the basin of attraction. I have choosen 36 initial points on the circle of radius 0.05. Here is the Matlab code:
axes
hold on
r= 0.05; % radius of the circle of initial points
rpm= @(t,x)[-x(1)+x(2)+x(1)*(x(1)^2+x(2)^2);...
-x(2)-2*x(1)+x(2)*(x(1)^2+x(2)^2)]; % the right part of the system
for fi= 0:pi/36:2*pi
% notice the backward in time direction of integration
[t,z]= ode45(rpm,15:-0.1:0,r*[cos(fi) sin(fi)]);
plot(z(:,1),z(:,2),'b');
end
grid on
h= ezplot('x^2+y^2=0.847^2',[-2 2],[-1.5 1.5]); % draw the circle
h.LineWidth= 1.7;
h.Color= 'red';
axis equal