Group with simple relations

If $b^{-1}ab=a^2$, $b^3 =1$ and $a^5=1$, then $$\begin{align*} b^{-2}ab^2 &= b^{-1}(b^{-1}ab)b\\ &= b^{-1}a^2b\\ &= (b^{-1}ab)^2\\ &= (a^2)^2 = a^4 = a^{-1}. \end{align*}$$ So therefore, $$\begin{align*} a&= b^{-3}ab^3\\ &= b^{-1}(b^{-2}ab^2)b\\ &= b^{-1}(a^{-1})b\\ &= (b^{-1}ab)^{-1}\\ &= (a^2)^{-1}\\ &= a^3. \end{align*}$$ Therefore, $a=a^3$.

Your original method fails because if we analyse what you have done:

We begin with $$b^{-1}ab=a^2$$

Pre-multiply by $b (=b^4)$ (**) $$ab=ba^2$$

Post-multiply by $a^3$ $$aba^3=b$$

Pre-multiply by $b^{-1}$ (*) $$b^{-1}aba^3=1$$

Post-multiply by $a^2$ $$b^{-1}ab=a^2$$

which is to be expected!

$aba^3=b$ is probably the best point to break, yielding $a^{-1}ba=a^{-2}ba^{-2}$, but Arturo's answer has already been accepted.