Square Matrix Inequality

There's a generalisation of Sylvester's Rank Inequality Theorem attributed to Frobenius. It states for all matrices $X, Y, Z$ we have $$ rk(XY) + rk(YZ) \le rk(Y) + rk(XYZ). $$ Using $AB = A + B$ we get $(A-I)(B-I)=I$ hence $(A-I) = (B-I)^{-1}$ and so $(B-I)(A-I) = I$ which implies $BA = A + B = BA$. So the matrices commute. Then apply the above theorem with $Y = AB$, $X = A-I$ and $Z = B-I$ gives the desired inequality.

Step 1: $B-I$ is invertible

In fact, $$ Bv = v \implies Av = ABv = Av + Bv = Av + v\implies v=0 $$

Step 2: $AB=BA$

In fact, since $B-I$ is invertible, in particular $(B-I)^{-1}=P(B)$ where $P$ is a polynomial (obtained by euclidean division between the characteristic polynomial and $x-1$). So $$ AB = A+B \implies A(B-I) = B \implies A = BP(B) = P(B)B $$ and $A$ is thus a polynomial in $B$. In particular $AB=BA$.

Step 3: Profit

Notice now that $A^2 = P(B)BA$, so $rk(A^2)\le rk(BA)$ and the problem is symmetric in $B,A$, so $rk(B^2)\le rk(AB)$, leading to $$ rk(A^2) + rk(B^2)\le rk(BA)+ rk(AB) = 2rk(AB) $$