How can I define a command that uses round parentheses around its arguments?

xparse makes defining a macro with a different kind of mandatory argument delimiter requirement fairly easy. Below, r() does just that.

\documentclass{article}

\usepackage{mathtools,xparse,etoolbox}

\DeclarePairedDelimiterX{\RoundBrackets}[1]{(}{)}{#1}

\NewDocumentCommand{\pr}{ r() }{%
  \def\prArg{#1}% Capture argument in macro
  \patchcmd{\prArg}{|}{\mid}{}{}% Replace | with \mid
  \RoundBrackets{\prArg}% Set argument in round brackets
}

\begin{document}

$\pr(a|b)$

\end{document}

etoolbox is used to replace | with \mid.

This also supports the usual options for \DeclarePairedDelimiter:

\documentclass{article}
\usepackage{amsmath}
\usepackage{xparse}
\usepackage{mleftright}

\ExplSyntaxOn

\NewDocumentCommand{\p}{sO{}r()}
 {
  \IfBooleanTF{#1}
   {
    \mleft(
    \danijar_middlevert:
    #3
    \mright)
   }
   {
    \group_begin:
    \danijar_sizedvert:n {#2}
    \mathopen{#2(}
    #3
    \mathclose{#2)}
    \group_end:
   }
 }

\cs_new_protected:Nn \danijar_middlevert:
 {
  \char_set_active_eq:NN | \__danijar_middle:
  \mathcode`|="8000 \scan_stop:
 }
\cs_new_protected:Nn \__danijar_middle:
 {
  \;\middle\vert\;
 }

\cs_new_protected:Nn \danijar_sizedvert:n
 {
  \tl_set:Nn \l__danijar_size_tl { #1 }
  \char_set_active_eq:NN | \__danijar_mid:
  \mathcode`|="8000 \scan_stop:
 }
\cs_new_protected:Nn \__danijar_mid:
 {
  \mathrel{\l__danijar_size_tl\vert}
 }

\ExplSyntaxOff

\begin{document}

$\p(x) \neq \p(x|y)$
\qquad
$\p[\big](x) \neq \p[\big](x|y)$
\qquad
$\p[\Big](x) \neq \p[\Big](x|y)$
\qquad
$\p*(\dfrac{a}{b})\neq \p*(\dfrac{a}{b}|y)$

\end{document}

The idea is to locally make | math active, with an appropriate definition, which is \;\middle\vert\; when automatic sizing is declared, or \mathrel{<size>\vert} when a manual size is selected.

If you want to add the “P” for “probability”:

\documentclass{article}
\usepackage{amsmath}
\usepackage{xparse}
\usepackage{mleftright}

\ExplSyntaxOn

\NewDocumentCommand{\p}{sO{}r()}
 {
  \operatorname{P}
  \IfBooleanTF{#1}
   {
    \mleft(
    \danijar_middlevert:
    #3
    \mright)
   }
   {
    \group_begin:
    \danijar_sizedvert:n {#2}
    \mathopen{#2(}
    #3
    \mathclose{#2)}
    \group_end:
   }
 }

\cs_new_protected:Nn \danijar_middlevert:
 {
  \char_set_active_eq:NN | \__danijar_middle:
  \mathcode`|="8000 \scan_stop:
 }
\cs_new_protected:Nn \__danijar_middle:
 {
  \;\middle\vert\;
 }

\cs_new_protected:Nn \danijar_sizedvert:n
 {
  \tl_set:Nn \l__danijar_size_tl { #1 }
  \char_set_active_eq:NN | \__danijar_mid:
  \mathcode`|="8000 \scan_stop:
 }
\cs_new_protected:Nn \__danijar_mid:
 {
  \mathrel{\l__danijar_size_tl\vert}
 }

\ExplSyntaxOff

\begin{document}

$\p(x) \neq \p(x|y)$

$\p[\big](x) \neq \p[\big](x|y)$

$\p[\Big](x) \neq \p[\Big](x|y)$

$\p*(\dfrac{a}{b})\neq \p*(\dfrac{a}{b}|y)$

\end{document}

Probably @Werner's answer is the way to go (robust and easily modified), but in this case, plain TeX also seems to work:

\documentclass{article}

\def\pr(#1|#2){(#1 \mid #2)}

\begin{document}

$\pr(a|b)$
$\pr(a_r|b^2)$

\end{document}