Dynamics of RSK

My original answer was completely wrong. But I think it was wrong in an interesting way; and I want to use this space to suggest that there is indeed a reasonable question to ask about the dynamics of RSK.

Let $\mathrm{Mat}_{n \times n}(\mathbb{Z})$ be the set of $n \times n$ matrices with entries in $\mathbb{Z}$. (Actually we can work with $\mathbb{Z}$-fillings of an arbitrary shape $\lambda$, but let's stick to square shapes here.) For $X = (x_{ij}) \in \mathrm{Mat}_{n \times n}(\mathbb{Z})$ define

• $\mathrm{rect}(X;i,j) := \sum_{i'\leq i, j'\leq j} x_{i',j'}$;
• $\mathrm{diag}(X;i,j) := \sum_{k=0}^{\mathrm{min}(i,j)-1} x_{i-k,j-k}$;
• $\mathrm{cohook}(X;i,j) := x_{i,j} + \sum_{k=1}^{i-1}x_{i-k,j} + \sum_{k=1}^{j-1} x_{i,j-k}$.

Let $\mathrm{DIAG}\colon \mathrm{Mat}_{n\times n}(\mathbb{Z}) \to \mathrm{Mat}_{n\times n}(\mathbb{Z})$ be the map that sends the matrix $X = (x_{ij}) \in \mathrm{Mat}_{n \times n}(\mathbb{Z})$ to the matrix $Y = (\mathrm{diag}(X;i,j))$. Note that $\mathrm{DIAG}$ is invertible as a map; let $\mathrm{DIAG}^{-1}$ denote this inverse. Define the maps of matrices $\mathrm{RECT}$ and $\mathrm{COHOOK}$ (and their inverses) similarly.

We can extend RSK to a map $\mathrm{RSK}\colon \mathrm{Mat}_{n\times n}(\mathbb{Z}) \to \mathrm{Mat}_{n\times n}(\mathbb{Z})$ by using the tropical toggle definition due to Pak/Berenstein-Kirillov as outlined in the notes linked to in the question. As pointed out in the question, however, naively studying iterates of $\mathrm{RSK}$ does not really make sense because the mass in the system is growing. So we want to renormalize. One way to do that is to define

$\varphi\colon \mathrm{Mat}_{n\times n}(\mathbb{Z}) \to \mathrm{Mat}_{n\times n}(\mathbb{Z}) := \mathrm{RECT}^{-1} \circ \mathrm{DIAG} \circ \mathrm{RSK}$

As Darij pointed out in the comments, $\mathrm{RECT}^{-1} \circ \mathrm{DIAG} = \mathrm{COHOOK}^{-1}$, so you can look at $\varphi$ that way if you want. At any rate, I believe the row and column sums of $X \in \mathrm{Mat}_{n\times n}(\mathbb{Z})$ and of $\varphi(X)$ agree, so that in particular the mass in the system is preserved under $\varphi$.

The $2 \times 2$ case looks like the following:

$\begin{pmatrix} a & b \\ c & d \end{pmatrix} \overset{\mathrm{RSK}}\mapsto \begin{pmatrix} \mathrm{min}(b,c) & a + b \\ a+c & a + d + \mathrm{max}(b,c) \end{pmatrix} \overset{\mathrm{DIAG}}\mapsto \begin{pmatrix} \mathrm{min}(b,c) & a + b \\ a+c & a + d + b + c \end{pmatrix} \overset{\mathrm{RECT}^{-1}}\mapsto \begin{pmatrix} \mathrm{min}(b,c) & a + b - \mathrm{min}(b,c) \\ a+c - \mathrm{min}(b,c) & d - a + \mathrm{min}(b,c) \end{pmatrix}$

So we have $\varphi^2 = \mathrm{id}$ when $n=2$. But already the $3 \times 3$ case exhibits interesting behavior. Let's look at the orbit of the identity map under $\varphi$:

1. $\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1\end{pmatrix} \overset{\mathrm{RSK}}\mapsto \begin{pmatrix} 0 & 0 & 1 \\ 0 & 0 & 2 \\ 1 & 2 & 3\end{pmatrix} \overset{\mathrm{COHOOK}^{-1}}\mapsto \begin{pmatrix} 0 & 0 & 1 \\ 0 & 0 & 1 \\ 1 & 1 & -1\end{pmatrix}$;
2. $\begin{pmatrix} 0 & 0 & 1 \\ 0 & 0 & 1 \\ 1 & 1 & -1\end{pmatrix} \overset{\mathrm{RSK}}\mapsto \begin{pmatrix} 0 & 0 & 1 \\ 0 & 2 & 2 \\ 1 & 2 & 1\end{pmatrix} \overset{\mathrm{COHOOK}^{-1}}\mapsto \begin{pmatrix} 0 & 0 & 1 \\ 0 & 2 & -1 \\ 1 & -1 & 1\end{pmatrix}$;
3. $\begin{pmatrix} 0 & 0 & 1 \\ 0 & 2 & -1 \\ 1 & -1 & 1\end{pmatrix} \overset{\mathrm{RSK}}\mapsto \begin{pmatrix} 1 & 1 & 1 \\ 1 & 0 & 1 \\ 1 & 1 & 2\end{pmatrix} \overset{\mathrm{COHOOK}^{-1}}\mapsto \begin{pmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0\end{pmatrix}$;
4. $\begin{pmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0\end{pmatrix} \overset{\mathrm{RSK}}\mapsto \begin{pmatrix} 0 & 0 & 1 \\ 0 & 1 & 2 \\ 1 & 2 & 3\end{pmatrix} \overset{\mathrm{COHOOK}^{-1}}\mapsto \begin{pmatrix} 0 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 0\end{pmatrix}$;
5. $\begin{pmatrix} 0 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 0\end{pmatrix} \overset{\mathrm{RSK}}\mapsto \begin{pmatrix} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1\end{pmatrix} \overset{\mathrm{COHOOK}^{-1}}\mapsto \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1\end{pmatrix}$.

So the order of $\varphi$ when $n=3$ is at least $5$. In fact, we also have for instance that

$\begin{pmatrix} 1 & 0 & 2 \\ 0 & 2 & 0 \\ 1 & 1 & 0 \end{pmatrix} \overset{\varphi}\leftrightarrow \begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 1 \\ 1 & 1 & 0 \end{pmatrix}$

so the order is at least $10$. The question of course is what is the order of $\varphi$ for arbitrary $n$; indeed, does it have finite order? What is its orbit structure?