Which finite group is this?

So $x=xy^3=yx^2y^2=y^2x^4y=y^3x^8=x^8$. As $x^3=1$ then $x=1$. The relations then collapse to $y^3=1$, and $G$ has order $3$ generated by $y$.

Another solution from relations:

You have the relation $xy=yx^2=yx^{-1}$. This means $y^{-1}xy=x^{-1}$. So $y$ conjugates $x$ to $x^{-1}$, and then $y^2$ conjugates $x$ to $(x^{-1})^{-1}=x$, and finally $y^3=1$ conjugates $x$ to $x^{-1}$. Thus $x=x^{-1}$, which is the same as saying $x^2=1$, and multiplying both sides by $x$ we get $1=x^3=x$.

EDIT: Here's a proof of my comment, that the group $G=\langle x,y\mid x^n=y^m=1, xy=yx^r\rangle$ has order $mn$ precisely when $r^m=1\pmod{n}$. I'll use the fact already given in the question body, that $G$ can have order at most $mn$ [this is seen by just listing all possible elements].

First, note that just like in my answer, we have $y^{-1}xy=x^r$, and thus $y^{-k}xy^k=x^{r^k}$. Letting $k=m$, we get

\begin{align} x &= y^{-m}xy^m\\ &= x^{r^m} \end{align} And thus $r^m-1$ divides $|x|$. So if $r^m\neq1\pmod{n}$, then $|x|\le\gcd(n, r^m-1)<n$, and so $|G|<mn$.

Now suppose $r^m=1\pmod{n}$. We can show $|G|=mn$ by giving an explicit (permutation) representation of $G$, that has order $mn$. [To see why, this shows $|G|\ge mn$, and together with $|G|\le mn$ we get $|G|=mn$.]

We will work in the symmetric group $S_{n+m}$, and let our representative for $x$ be the $n$-cycle $(1, 2, \ldots, n)$. Note that $x^r$ then looks like $(1, r+1, 2r+1, \ldots)$. If we define the permutation $\alpha$ as $$ \begin{pmatrix} 1 & 2 & \cdots & n\\ 1 & r+1 & \cdots & (n-1)r+1 \end{pmatrix}$$ then we have $\alpha^{-1}x\alpha=x^r$. In fact, we can give an explicit formula for $\alpha$: $$ \alpha(i) = (i-1)r+1\pmod{n}$$

Note that, for any positive integer $k$, we get $\alpha^k(i)=(i-1)r^k+1\pmod{n}$. Thus, once again letting $k=m$, we have $$ \alpha^m(i) = (i-1)r^m+1\pmod{n}$$ Since $r^m=1\pmod{n}$, we see then that $\alpha^m(i)=i$ for all $i\in\{1, \ldots,n\}$.

Thus $|\alpha|$ divides $m$. Unfortunately, we're not quite done, since it's not always true $\alpha$ has order $m$ (to see an explicit example, try $n=m=4$ and $r=3$).

However, this is why we work in $S_{n+m}$, and not just $S_n$. We can take the representative for $y$ to be $\alpha\cdot(n+1, \ldots, n+m)$, and then $|y|=m$ and $y$ acts on $x$ the same way $\alpha$ does: $y^{-1}xy=x^r$.

Finally, if $H=\langle x,y\rangle\le S_{n+m}$, then $K=\langle x\rangle$ is normal in $H$, and $H/K\cong\langle y\rangle$. Since $|x|=n$ and $|y|=m$, we have $|H|=mn$, as desired.

Any group of order $P$ or $p^2$ is Abelian. So, $xy = yx^2$ implies, post computation that order is 3 or 9, that $x = 1$