Let $A,B,C$ be $n \times n$ square real matrices with $ABC=0$. What is the maximum rank of $CBA$?

First, note that $\dim(\ker(ABC)) \le \dim(\ker(A))+\dim(\ker(B))+\dim(\ker(C))$, so at least one of the three kernels is at least $\lceil n/3\rceil$. Next, note that $\textrm{rank}(CBA)\le \min\{\textrm{rank}(C),\textrm{rank}(B),\textrm{rank}(A)\}$, so we can conclude $\textrm{rank}(CBA) \le n-\lceil n/3\rceil = \lfloor 2n/3 \rfloor$.

We now demonstrate that this bound can actually be obtained. Write $n = 3k+l+m$, where $0\le l \le m\le 1$. Note that $2k+l = \lfloor 2n/3 \rfloor$. Now we label a basis of $\Bbb R^n$ as $x_1, x_2, \ldots, x_{k+m}, y_1, \ldots, y_{k+l}, z_1, \ldots, z_k$.

Define the maps $A,B,C$ on basis elements as follows:

• $Ax_i = 0, Ay_i = y_i, Az_i = z_i$
• $Bx_i = x_i, By_i = x_i, Bz_i = z_i$
• $Cx_i = x_i, Cy_i = y_i, Cz_i = y_i$

Then we have $ABC = 0$, but $CBAy_i = x_i, CBAz_i = y_i$, so the rank of $CBA$ is equal to the number of $y_i$'s and $z_i$'s, ie $2k+l = \lfloor 2n/3 \rfloor$.

The usual rank inequality gives $$ rk(CBA)\le {\rm min}\{rk(A),rk(B),rk(C)\} $$ for all $A,B,C$. Because in addition $ABC=0$ we can use the Frobenius rank inequality to obtain $$ rk(A)+rk(B)+rk(C)\le 2n. $$ Hence the upper bound for $rk(CBA)$ is at most $\lfloor 2n/3 \rfloor$, if $A$, $B$ and $C$ have the same rank. It is easy to see that this bound can be attained.

In general, the nullity of a product $\DeclareMathOperator{\rank}{rank}AB$ satisfies $$\max\{\dim \ker A, \dim \ker B\} \leq \dim \ker (AB) \leq \dim \ker A + \dim \ker B.$$ In our case we have $$n = \dim \ker (ABC) \leq \dim \ker A + \dim \ker B + \dim \ker C,$$ so at least one of $A, B, C$ has kernel of dimension $\geq \left\lceil \frac{n}{3} \right\rceil$. Thus, $$\ker (CBA) \geq \max\{\ker A, \ker B, \ker C\} \geq \left\lceil \frac{n}{3} \right\rceil$$ and hence by the Rank-Nullity Theorem, $$\rank (CBA) \leq n - \left\lceil \frac{n}{3} \right\rceil = \left\lfloor \frac{2n}{3} \right\rfloor.$$ We can prove that this bound is sharp by exhibiting a triple $(A, B, C)$ that achieves it. We can see that it's enough find matrices $\alpha, \beta, \gamma$ that respectively solve the problem for $n = 3$. Then, for any $n$, we can set $$A = \underbrace{\alpha \oplus \cdots \oplus \alpha}_{\left\lfloor\frac{n}{3}\right\rfloor} \oplus 0 ,$$ where $0$ is the $0 \times 0$ matrix of size $r \times r$, where $r$ is the remainder of $n$ modulo $3$, and define $B$ and $C$ similarly. I don't know a good way to find a $3 \times 3$ solution except by trial and error. A short script that tries random $(0, 1)$-matrices yields the pleasant-looking solution $$\alpha = \pmatrix{0&1&0\\0&0&1\\0&0&0}, \qquad \beta = \pmatrix{0&1&0\\1&0&0\\0&0&0}, \qquad \gamma = \pmatrix{0&0&0\\0&1&0\\1&0&0} .$$