Misplaced alignment tab character when building a matrix

The matrix macro is defined as part of the basic LaTeX distribution, but you're probably after the amsmath implementation of it:

\documentclass{article}
\usepackage{amsmath}% http://ctan.org/pkg/amsmath
\begin{document}
\[
\begin{matrix}
5 & 6 & 7 \\
4 & P & 0 \\
3 & 2 & 1
\end{matrix}
\]
\end{document}

The older (TeX) \matrix equivalent actually looks like this:

\documentclass{article}
\begin{document}
\[
\matrix{%
5 & 6 & 7 \cr
4 & P & 0 \cr
3 & 2 & 1
}
\]
\end{document}

Note that \matrix is a macro taking a single argument, and not an environment.

Standard LaTeX does not have a matrix environment. For historical reasons the original version of LaTeX (LaTeX 2.09) contained a more or less full copy of plain TeX commands including a \matrix command. Some of these plain TeX commands made little to no sense in LaTeX as they used quite different conventions (and some of them didn't even work at all with LaTeX such as \topinsert or \beginsection). Some made sense and they got documented in the LaTeX manual as "LaTeX commands".

So when LaTeX2e (the current version of LaTeX) was designed, we took those that did not work out. However, those that did work (even if not documented and with unusual syntax) have been kept for compatibility reasons, just in case somebody did use them in old documents. E.g.,

\maxtrix{5 & 6 & 7 \cr
        4 & P & 0 \cr
        3 & 2 & 1 \cr}

should work in plain TeX (and in fact in LaTeX). But neither \cr nor \matrix is a documented and official LaTeX command. And as you can see it is a command with one argument not an environment, which is why you get your error message.

In LaTeX it is best to load the amsmath package that provides all kind of good math support including a matrix environment that supports the syntax you used.