Rank of $X\otimes Y-Y\otimes X$

Consider the generic case; for example, randomly choose $X,Y\in M_n$.

$(*)$ Then $\ker(X\otimes Y-Y\otimes X)$ is a subspace of the symmetric tensors of order $2$; in particular, it admits a basis of (eventually complex) vectors of the form $v\otimes v$ where $v\in\mathbb{C}^n$.

$\textbf{Proposition}$. Generically, $dim(\ker(X\otimes Y-Y\otimes X))=n$ and, consequently, $rank(\ker(X\otimes Y-Y\otimes X))=n^2-n$.

$\textbf{Proof}$. Assume that $(X\otimes Y-Y\otimes X)(v\otimes v)=0$; then $X(v)\otimes Y(v)=Y(v)\otimes X(v)$, that implies that there is $\lambda\in\mathbb{C}$ s.t. $X(v)=\lambda Y(v)$; thus $\lambda$ is a root of $\det(X-\lambda Y)=0$. Generically, the previous polynomial has $n$ distinct complex roots and the kernel associated to each root $\lambda$ has dimension $1$ (that is, there is exactly one normalized vector $v\otimes v$ -in the kernel-, associated to each root $\lambda$). $\square$

EDIT. The assertion $(*)$ is only valid when $X,Y$ are generic (otherwise, choose, for example, $X=0$ or $X=Y$). To prove $(*)$, it suffices to show that if $u,v$ are vectors s.t.

$(**)$ $(X\otimes Y-Y\otimes X)(u\otimes v-v\otimes u)=0$, then $u,v$ are parallel.

I didn't write the proof, but the idea is as follows

We assume that $u,v$ is a free system. Then $(**)$ implies the existence of algebraic relations linking the entries $(x_{i,j},y_{i,j})$. Then the set of $X,Y$ satisfying $(**)$ is Zariski closed, that is contradictory with the hypothesis of genericity.

For example, when $n=2$, there are $3$ relations

Some observations:

• Via the vectorization operator, we can think of this as being the map $$ M \mapsto YMX^T - XM Y^T. $$
• If $Y = \alpha X$ for some scalar $\alpha$, then $A = 0$.
• If $Y = I$ (the identity matrix) and $A$ is diagonalizable, then the kernel of $A$ is spanned by vectors of the form $v \otimes w$ where $Xv = \lambda v$ and $Xw = \lambda w$. $A$ will have rank $n^2 - n$ if $X$ has no repeating eigenvalues and smaller rank otherwise.
• If $Y$ is invertible, we can reduce to the $Y=I$ case by noting that $$ (Y \otimes Y)^{-1}A = (Y^{-1}X) \otimes I - I \otimes (Y^{-1}X). $$
• In general, we can guarantee that $\operatorname{rank}(A) \leq 2\operatorname{rank}(X)\operatorname{rank}(Y)$.