Inverse of block anti-diagonal matrix
I think this is the answer with all the blocks invertible. $$ A = \begin{pmatrix} & & & A_1 \\ & & A_2 & \\ & \dots & & \\ A_d\end{pmatrix}, $$
$$ B = \begin{pmatrix} & & & A_d^{-1} \\ & & A_{d-1}^{-1} & \\ & \dots & & \\ A_1^{-1}\end{pmatrix}, $$ we have
$$AB=I$$
There exists a permutation matrix $\rm P$ such that
$${\rm A P} = \mbox{diag} \left( {\rm A}_1, {\rm A}_2, \dots, {\rm A}_d \right)$$
Assuming that all the ${\rm A}_i$ blocks are invertible,
$$\left( \rm A P \right)^{-1} = {\rm P}^\top {\rm A}^{-1} = \mbox{diag} \left( {\rm A}_1^{-1}, {\rm A}_2^{-1}, \dots, {\rm A}_d^{-1} \right)$$
and, thus,
$${\rm A}^{-1} = \color{blue}{{\rm P} \, \mbox{diag} \left( {\rm A}_1^{-1}, {\rm A}_2^{-1}, \dots, {\rm A}_d^{-1} \right)}$$
For example, if $d = 3$,
$${\rm A}^{-1} = \begin{bmatrix} & & {\rm I}\\ & {\rm I} & \\ {\rm I} & & \end{bmatrix} \begin{bmatrix} {\rm A}_1^{-1} & & \\ & {\rm A}_2^{-1} & \\ & & {\rm A}_3^{-1}\end{bmatrix} = \begin{bmatrix} & & {\rm A}_3^{-1}\\ & {\rm A}_2^{-1} & \\ {\rm A}_1^{-1} & & \end{bmatrix}$$
linear-algebra matrices block-matrices permutation-matrices