Invertible matrices satisfying $[x,y,y]=x$

The answer is "No". Indeed, consider the 1-related group $G=\langle x,y \mid [x,y,y]=x\rangle$ That group has a presentation $\langle a,b,t \mid a^t=ab, b^t=ba\rangle$ (easy to check). Thus it is an ascending HNN extension of the free group. The group $G$ is hyperbolic (proved by Minasyan using the Bestvina-Feighn combination theorem). By a theorem of Hagen and Wise, the group (and almost every other hyperbolic ascending HNN extensions of a free group) acts geometrically on a finite dimensional CAT(0)-cube complex. By a result of Ian Agol then the group $G$ is virtually special and hence linear, i.e., there exists an injective homomorphism $\phi$ from $G$ to a special linear group (over $\mathbb{Z}$). Since $G$ is hyperbolic, it does not contain nilpotent non-Abelian subgroups. Thus the pair of integer matrices $(\phi(a), \phi(b))$ is an example showing that the answer is "no".

Here's a quick test which might disprove your hopes very quickly:

Take $n$ to be small: Try $2$ first, and $5$ is probably near the limit of a computer algebra system. Choose $x$ to be a random $n \times n$ diagonal matrix with determinant $1$, for example, $\mathrm{diag}(17, 1/17)$. Write out your relation, leaving all the elements of $y$ as variables. After clearing denominators, you have $n^2$ simultaneuous homogenous equations in $n^2$ variables. (If I haven't made any dumb errors, they have degree $3n$.) Ask your favorite computer algebra system to solve them for you. If any of the roots are not on the hypersurface $\det y=0$, then you have a counterexample!