Show that the equilibrium point $(0,0)$ is asymptotically stable and an estimate of its basin of attraction

In addition to what has been said by @Evgeny and @MrYouMath: the set $$ M=\left\{ (x,y)\in\mathbb R^2 :\; x^2+y^2<2 \right\} $$ is a positively invariant set of the considered system since $\forall (x,y)\in M$ $$ \dot V=-x^4-y^4+x^2y^2(x^2+y^2)<-x^4-y^4+2x^2y^2=-(x^2-y^2)^2\leq 0; $$ it is also a subset (guaranteed estimation) of the domain of attraction.


As @Evgeny suggested you could use the Lyapunov function candidate

$$V(x,y)=\dfrac{1}{2}\left[x^2+y^2\right].$$

$V(x,y)$ is clearly positive definite at the origin and radially unbounded (which we would need for assessing global asymptotic stability).

The time derivative of $V(x,y)$ is given by $$\dot{V}=x\dot{x}+y\dot{y}=x(-y-x^3+x^3y^2)+y(x-y^3+x^2y^3)$$ $$\dot{V}=-x^4-y^4+x^2y^2(x^2+y^2)=-(1-y^2)x^4-(1-x^2)y^4.$$

As the lower order terms $-x^4-y^4$ are negative definite we can conclude that the equilibrium point is asymptotically stable in a region around the origin. Using the comment given by @Evgeny we can see that this expression is negative semidefinite if $(x,y)$ lie inside the unit circle $D=\{(x,y)\in \mathbb{R}^2|x^2+y^2<1\}$. This is the basin of attraction (correction due to @Artem).

We cannot say if the origin is globally asymptotically stable because $\dot{V}$ is not negative definite for all regions around the origin. This does not mean that the origin cannot be globally asymptotically stable. It just means we can only show (local) asymptotic stability of the origin with $V(x)$ as our Lyapunov function candidate.