Mutual capacitance upper limit

Yes, there is a constraint also for the mutual capacitance. As I shortly reviewed in this answer, the capacitance matrix of a system of conductors is a symmetric and positive definite matrix, exactly as the inductance matrix of a system of inductors. Since the relationship $M\le \sqrt{L_1L_2}$ can be directly obtained from this property, which, in turns can be derived from the conservation of energy, it's not difficult to prove the analogue for a system of three conductors where one is taken as reference conductor (ground).

Consider the system of two conductors with ground shown below (beware the sign of $C_{12}$: what you called "mutual capacitance" is actually $-C_{12}$, $C_{12}=C_{21}$ being the off-diagonal element of the capacitance matrix):

For such a system, the capacitance matrix is a 2-by-2 symmetric matrix with $C_{11} = C_1-C_{12}$, $C_{22} = C_2-C_{12}$ and $C_{12}<0$ ($C_{11}$ and $C_{22}$ are the total capacitances from each conductor to the other conductors). Such a matrix is further positive definite if and only if its trace and determinant are positive (see e.g. this question on Math SE). From the condition on the determinant, we thus have

$$C_{11}C_{22}-C_{12}^2\ge 0,$$

that is,

$$|C_{12}|\le\sqrt{C_{11}C_{22}}.$$

For further details and derivations for the general case, see:

R. M. Fano, L. J. Chu, and R. B. Adler, Electromagnetic fields, energy, and forces, MIT Press, 1968.