Question about the basin of attraction of the origin

Hints: Instead of considering $x^2+2y^2+z^2$ it is in fact easier to consider $z$ separately from $(x,y)$.

1) Integrate the equation for $z$ and show that $z(t)$ goes to zero as $t\rightarrow +\infty$.

2) Show that the function: $L(t)=x(t)^2+2y(t)^2$ verifies the ode: $\dot{L}(t) =-2|\epsilon| L(t) \; (z(t)+1).$

3) Integrate this ode for $L$ and show that $L(t)\rightarrow 0$ as $t\rightarrow +\infty$.

(Along the way you should also conclude that neither $L$, nor $z$ can go to infinity)

I'm quite sure that this is a problem from Hirsch, Smale, and Devaney's book in Dynamical systems. So let's take it piece by piece.

a) When $\epsilon = 0$ we have that

$$x ' = 2y(z+1)$$ $$y' = -x(z+1)$$ $$z' = -z^3$$

Now we want to show that the origin is not asymptotically stable. This means there exists some solution that does not tend towards the origin as $t \to \infty$ but stays close to the origin (being a little imprecise here, but you can make it precise by looking at the definition of stable).

Let's try to construct such a solution. For simplicity, let's make the bold assumption that $z = 0$, in which case we now have the equations

$$x' = 2y$$ $$y' = -x$$ $$z' = 0$$

This is basically a $2 \times 2$ system with matrix

$$\begin{bmatrix} 0 & 2 \\ -1 & 0 \\ \end{bmatrix}$$

which has eigenvalues $\pm i \sqrt{2}$. Thus we have solutions to this system of form

$$\begin{bmatrix} x \\ y \end{bmatrix} = C_1 \begin{bmatrix} \cos{t\sqrt{2}} \\ -\sin{t\sqrt{2}} \end{bmatrix} + C_2 \begin{bmatrix} \sin{t\sqrt{2}} \\ \cos{t\sqrt{2}} \end{bmatrix} $$

Putting it together,

$$\begin{bmatrix} x \\ y \\ z \end{bmatrix} = C_1 \begin{bmatrix} \cos{t\sqrt{2}} \\ -\sin{t\sqrt{2}} \\ 0\end{bmatrix} + C_2 \begin{bmatrix} \sin{t\sqrt{2}} \\ \cos{t\sqrt{2}} \\ 0\end{bmatrix} $$

which are ellipses around the origin. These are stable but not asymptotically stable.

b) Here, we can use LaSalle's Invariance Principle. On page 195 (section 9.2 of Hirsch, Smale, and Devaney) we saw that

$$L = x^2+2y^2+z^2$$

is a Liapunov function. Further, we saw that it is a strict Liapunov function since

$$\frac{dL}{dt} = \epsilon(x^2+2y^2)(z+1)-z^4 < 0$$

for $\epsilon < 0$ and $z > -1$.

LaSalle's Invariance Principle given in the textbook says:

Let $X^*$ be an equilibrium point for $X'= F(X)$ and let $L: U \to \mathbb{R}$ be a Liapunov function for $X^*$, where $U$ is an open set containing $X^*$. Let $P \subseteq U$ be a neighborhood of $X^*$ that is closed and bounded. Suppose that $P$ is positively invariant, and that there is no entire solution in $P\setminus\{X^*\}$ on which $L$ is constant. Then $X^*$ is asymptotically stable and $P$ is contained in the basin of attraction of $X^*$.

Here's what we will do. Consider the set

$$P_{abc} = \{(x,y,z) \in \mathbb{R}^3 \quad | \quad |x| \le a, |y| \le b, -1+c \le z \le b$$

Notice that

$$\bigcup_\limits{a > 0, b > 0, 0<c<1} P_{abc} = \{(x,y,z) \in \mathbb{R}^3 \quad | z > -1 \}$$

Furthermore, we have that each $P_{abc}$ contains the origin and is closed. Also, since the Liapunuv function is strict, then each $P_{abc}$ is positively invariant else it would imply that the Liapunov function is increasing (WARNING: THIS IS FALSE. As noted by a comment below, it is NOT true that each of these $P_{abc}$ are positively invariant. I am leaving this solution up here so that you can be warned of my error). Furthermore, having a strict Liapunov function also guarantees that $L$ is not constant along a solution. Thus by LaSalle's Invariance Principle, we must have that each $P_{abc}$ is in the basin of the origin and so the union is also in the basin.

I will add a more detailed answer to the hint given by @H. H. Rugh (credit goes to him).

As H. H. Rugh proposed you need to find a Lyapunov function and the proposed function works. So we start by using

$$L(x,y,z)=x^2+2y^2+z^2$$

as a positive definite function, which is only zero for the origin (which is an equilibrium of the system).

The time derivative is given by

$$\dfrac{d}{dt}L(x,y,z)=2x\dot{x}+4y\dot{y}+2z\dot{z}$$ $$=2x(\varepsilon x+2y)(z+1)+4y(\varepsilon y-x)(z+1)+2z(-z^3)$$ $$=-2(-\varepsilon)(z+1)(x^2+2y^2)-2z^4.$$

As can be easily seen by the last equation we can see that the time derivative is negative definite as long as $\varepsilon<0$ and $z+1>0 \implies z>-1$.

In order to estimate the region of attraction, we observe that $z>-1$ and $L(x,y,z)=x^2+2y^2+z^2<c$ in which we need to find the largest $c \in \mathbb{R}$ such that it does not violate $z>-1$. The Set $x^2+2y^2+z^2<c$ is an ellipsoid which is squeezed in the $y$-direction by a factor of 2 (draw a picture of the situation to conclude the following statement). Hence, we can conclude that $c=1$ qualifies for the region of attraction.