Existence of periodic solution in an ODE system

You can consider the curve $\phi = (x, y)$ and take the squared distance to the origin $x^2 + y^2$. The derivative of this is $$2(x\dot{x} + y\dot{y}) = 2(x^4 + x^6 + y^2 + y^8)\ge 0.$$ If this curve is periodic, then it is actually a closed curve and so a compact subset of the plane so there must be a minimum of this distance, corresponding to a point closest to the origin. Since it is a sum of squares this means $x = y = 0$ at that point, so your curve goes through the origin, but clearly the constant curve $\phi\equiv 0$ satisfies the equation and it can be the only one doing that.

If you wanted to conclude this from Stokes Theorem you need to integrate the 1-form $$(x^3 + x^5)dx + (y + y^7)dy$$ along the line given by a potential closed curve, i.e. around $\phi$ and if $D$ is the interior, assuming $T$ is the minimum period so that you avoid self intersections, you conclude $$\int_C (x^3 + x^5)dx + (y + y^7)dy = \int\int_D \dfrac{d(y + y^7)}{dx} - \dfrac{d(x^3 + x^5)}{dy} dxdy = 0.$$ But, by definition of evaluating a line integral, this is the same as $$\int_C (x^3 + x^5)dx + (y + y^7)dy = \int \phi'(t)\cdot\phi'(t) dt,$$ where we have used that $\phi$ satisfies the differential equation to get the first $\phi'$. The integrand is nonnegative and since it integrates to $0$, it is equal to $0$ everywhere in the interval it is defined. That is, $\phi \equiv 0$.

Hint: $x^3 + x^5 > 0$ when $x > 0$, $< 0$ when $x < 0$. Similarly for $y + y^7$.