Optional argument to an environment inside align

put the mat environment into braces:

\begin{align*}
 { \begin{mat}
           &\vdots                    \\
    \cdots &ra_{j,i}+sb_{j,i} &\cdots \\
           &\vdots
  \end{mat} }
  &= 0
\end{align*}

It's an expansion time problem. A way out comes from xparse:

\usepackage{mathtools,xparse}
\NewDocumentEnvironment{mat}{O{c}}
  {\begin{pmatrix*}[#1]}
  {\end{pmatrix*}}

Solution without xparse

Redefine \env@matrix, but not with \renewcommand:

\makeatletter
\def\env@matrix{\new@ifnextchar[\env@m@trix{\env@m@trix[c]}}
\def\env@m@trix[#1]{%
 \hskip-\arraycolsep\let\@ifnextchar\new@ifnextchar\array{*\c@MaxMatrixCols#1}}
\makeatother

This will allow you to use an optional argument to the Xmatrix environments; however you can't define an abbreviation with \newenvironment. This will work (with or without the optional argument)

\begin{align*}
  \begin{pmatrix}[r]
           &\vdots                    \\
    \cdots &ra_{j,i}+sb_{j,i} &\cdots \\
           &\vdots
  \end{pmatrix}
  &= 0
\end{align*}

If you want an abbreviation, then the magic formula

\expandafter\let\expandafter\mat\csname pmatrix*\endcsname
\expandafter\let\expandafter\endmat\csname endpmatrix*\endcsname

will allow you to write

\begin{align*}
  \begin{mat}
           &\vdots                    \\
    \cdots &ra_{j,i}+sb_{j,i} &\cdots \\
           &\vdots
  \end{mat}
\end{align}

with or without the optional argument.

Solution with minimal impact

(Thanks to Bruno Le Floch)

Write

\begin{mat}
\relax & \vdots                    \\
\cdots & ra_{j,i}+sb_{j,i} &\cdots \\
       & \vdots
\end{mat}