Inverse of product of matrices

If $ABA^T$ is invertible then $A(BA^T(ABA^T)^{-1})=I_m$, that is, $A$ is right-invertible. Similarly $A^T$ is left-invertible with left-inverse $(ABA^T)^{-1}AB$.

Write, abusing notation, $A^{-1}$ and $A^{-T}$ for the right- and left inverses respectively, and $C$ for $ABA^T$. Then $C^{-1}=(C^{-1}AB)B^{-1}(BA^TC^{-1})=A^{-T}B^{-1}A^{-1}$: the same formula holds true.

I am going to give an answer only for a symmetric positive definite $B$.

In this case the answer is

$$ (ABA^{\rm T})^{-1} = A^{+ \rm T} B^{-1/2} XB^{-1/2}A^+, $$ where $$ X = I-B^{-1/2}(I-A^{+}A) \big( B^{-1/2}(I-A^{+}A)\big)^+ $$

and ${}^+$ stands for the Moore-Penrose inverse.

---

One way of derivation is as follows.

Denote

$$ F = B^{-1/2}, \quad \quad G = I-A^{+}A, \quad \quad H(\alpha) = \Big(A(\alpha I +FF)^{-1}A^{\rm T}\Big)^{-1}. $$

Note that

$$ (ABA^{\rm T})^{-1} = \lim_{\alpha \to 0} \Big(A(\alpha I +FF)^{-1}A^{\rm T}\Big)^{-1} = \lim_{\alpha \to 0} H(\alpha). $$

Use Woodbury matrix identity to rewrite $H(\alpha)$ as

$$ H(\alpha) = \Big( \frac{1}{\alpha}AA^{\rm T}-\frac{1}{\alpha}AF(I+\frac{1}{\alpha}FF)^{-1}\frac{1}{\alpha}FA^{\rm T}\Big)^{-1} . $$

Note that $AA^{\rm T}$ is invertible and use Woodbury matrix identity again to obtain

$$ H(\alpha) = \alpha (AA^{\rm T})^{-1} - (AA^{\rm T})^{-1} AF \Big( -(I+\frac{1}{\alpha}FF) + \frac{1}{\alpha}FA^{\rm T}(AA^{\rm T})^{-1}AF \Big)^{-1} FA^{\rm T} (AA^{\rm T})^{-1}, $$

$$ H(\alpha) = \alpha (AA^{\rm T})^{-1} + (AA^{\rm T})^{-1} AF \Big( I+\frac{1}{\alpha}F \big(I - A^{\rm T}(AA^{\rm T})^{-1}A \big) F \Big)^{-1} FA^{\rm T} (A A^{\rm T})^{-1}. $$

Note that

$$ A^{\rm T} (A A^{\rm T})^{-1} = A^+, \quad \quad I - A^{\rm T}(AA^{\rm T})^{-1}A = G = G^2, $$

and rewrite $H(\alpha)$ as

$$ H(\alpha) =\alpha (AA^{\rm T})^{-1} + A^{+ \rm T} F \Big( I + \frac{1}{\alpha}FGGF\Big)^{-1}FA^+. $$

Use Woodbury matrix identity again to get

$$ H(\alpha) =\alpha (AA^{\rm T})^{-1} + A^{+ \rm T} F \Big( I-FG \big( \alpha I + GFFG \big)^{-1} GF \Big)FA^+. $$

Take the limit as $\alpha \to 0$ and apply the limit relation to obtain

$$ (ABA^{\rm T})^{-1} = A^{+ \rm T} F \Big( I-FG \big( FG \big)^+ \Big)FA^+. $$

Substitute $F$ and $G$ to get the final answer.