Integer matrices with integer inverses

Exactly those whose determinant is $1$ or $-1$.

See the previous question about the $2\times 2$ case. The determinant map gives necessity, the adjugate formula for the inverse gives sufficiency.

The inverse of an integer matrix is again an integer matrix iff if the determinant of the matrix is $\pm 1$. Integer matrices of determinant $\pm 1$ form the General Linear Group $GL(n,\mathbb{Z})$

Arturo and Sivaram have already given the general condition for integer matrices with integer inverses; here I only note this particular example due to Ericksen that the matrix $\mathbf A$ with entries

$$a_{ij}=\binom{n+j-1}{i-1}$$

where $n$ is an arbitrary nonnegative integer has an integer inverse.