Table inside enumeration: How to align vertically when using booktabs' \toprule?

You need to forcefully adjust the first line to align with the label. Below I've introduced ttabular that inserts this vertical adjustment (comprised of a regular baseline skip plus all the rule widths that make up \toprule^*). Horizontal centering is obtain using an \hfill<item>\hfill\null approach. You can also use a box of width \linewidth.

\documentclass{article}
\usepackage{enumitem,booktabs}% http://ctan.org/pkg/{enumitem,booktabs}

\setenumerate{listparindent=\parindent, parsep=0pt}

\newenvironment{ttabular}[2][c]
  {\begin{tabular}[#1]{#2}
     \\[\dimexpr-\normalbaselineskip-\abovetopsep-\belowrulesep-\heavyrulewidth]
  }{\end{tabular}}

\begin{document}
\begin{enumerate}[align=left, label=\large\sffamily\bfseries \alph*)]
  \item \hfill%
  \begin{ttabular}[t]{cccc}
    \toprule
    $m$ & $P(R=a,M=m)$ & $P(R=b,M=m)$ & $P(R=c,M=m)$ \\
    \midrule
    1 & 0.1 & 0.2 & 0.3 \\
    2 & 0.2 & 0.1 & 0.1 \\
    \bottomrule
  \end{ttabular}%
  \hfill\null%

  \item
  \( 
    \displaystyle
    P(0) = P(0|B) P(B) + P(0|G) P(G)
  \)
\end{enumerate}
\end{document}

Note that the vertical adjustment causes the tabular to rise quite high, which may overlap with contents above it. In that sense, you'll have to make adjustments to accommodate for that as well, if needed.

^* One of these lengths \abovetopsep is actually set to 0pt by default.

You can emulate what array does with \firsthline; however, the tabular will most likely clash with the preceding material and you have to add some vertical space anyway:

\documentclass{article}
\usepackage{lipsum} % just for the example
\usepackage{enumitem}
\usepackage{booktabs}

\makeatletter
\newcommand{\firsttoprule}{%
  \addlinespace[\dimexpr-\aboverulesep-\belowrulesep-\heavyrulewidth-\ht\@arstrutbox]
  \toprule}
\makeatother

\setlist[enumerate]{listparindent=\parindent, parsep=0pt}

\begin{document}
\lipsum*[2]
\begin{enumerate}[
  align=left,
  label=\large\sffamily\bfseries\alph*),
  before=\vspace{1.5ex}
]
\item
  \begin{tabular}[t]{cccc}
  \firsttoprule
  $m$ & $P(R=a,M=m)$ & $P(R=b,M=m)$ & $P(R=c,M=m)$ \\
  \midrule
  1 & 0.1 & 0.2 & 0.3 \\
  2 & 0.2 & 0.1 & 0.1 \\
  \bottomrule
  \end{tabular}

\item
  \( 
    \displaystyle
    P(0) = P(0|B) P(B) + P(0|G) P(G)
  \)
\end{enumerate}
\lipsum[2]
\end{document}

Note that \setenumerate is a deprecated command.

You may also use the adjustbox package by adjusting the vertical alignment:

\documentclass{article}
\usepackage{enumitem}
\usepackage{booktabs}
\usepackage{adjustbox}

\setlist[enumerate]{listparindent=\parindent, parsep=0pt} % edited as per @egreg comment

\begin{document}
\begin{enumerate}[align=left, label=\large\sffamily\bfseries \alph*)]
  \item
    \begin{adjustbox}{valign=t} % The key value, t, adjusts the vertical alignment
    \makebox[\linewidth]{       % Causes the centering
    \begin{tabular}{cccc}
    %\noalign{\vskip-.85ex}      % Uncomment to see the second output
    \toprule
    $m$ & $P(R=a,M=m)$ & $P(R=b,M=m)$ & $P(R=c,M=m)$ \\
    \midrule
    1 & 0.1 & 0.2 & 0.3 \\
    2 & 0.2 & 0.1 & 0.1 \\
    \bottomrule
  \end{tabular}}
    \end{adjustbox}
  \item
  \( 
    \displaystyle
    P(0) = P(0|B) P(B) + P(0|G) P(G)
  \)
\end{enumerate}
\end{document}