Matrix Inverse symbol

If you just want to reduce the size of the -1 term, you could create your own macro called, say, \inv{...}, along the lines of the code below. The screenshot shows the look of both \inv{X} and X^{-1}. (The vertical offset of 1.15ex is chosen so that the tops of the 1 symbols are at very nearly the same height using either \inv{...} or X^{-1}. Feel free to adjust the offset to suit your preferences.)

\documentclass{article}
\newcommand\inv[1]{#1\raisebox{1.15ex}{$\scriptscriptstyle-\!1$}}
\begin{document}
$X \inv{X} X^{-1}$
\end{document}

I see nothing bad in the usual notation X^{-1}. Here's a possible redefinition

\documentclass{article}
\usepackage{amsmath}

\newcommand{\inv}{^{\raisebox{.2ex}{$\scriptscriptstyle-1$}}}

\begin{document}
\[
X^{-1}\ne X\inv
\]
\end{document}