\left( and \right) too big when surrounding a sum (etc.) with no superscript; exacerbated by substack

Use \Big or \bigg; my preference would go to the former. It's not necessary that the parentheses encompass the whole contents, they are just delimiters. Surely having them asymmetric with respect to the formula axis would be worse.

\documentclass{article}
\usepackage{amsmath,amssymb}
\begin{document}
\[
\varphi^{-1}\left(\bigcap_{\substack{\mathfrak{p} < B \\ p \text{ prime}}} \mathfrak{p}\right)
\]
\[
\varphi^{-1}\Bigl(\bigcap_{\substack{\mathfrak{p} < B \\ p \text{ prime}}} \mathfrak{p}\Bigr)
\]
\[
\varphi^{-1}\biggl(\bigcap_{\substack{\mathfrak{p} < B \\ p \text{ prime}}} \mathfrak{p}\biggr)
\]
\end{document}

Compare with

\[
\varphi^{-1}
\mathopen{\raisebox{-1.8ex}{$\Biggl($}}\,
\bigcap_{\substack{\mathfrak{p} < B \\ p \text{ prime}}} \mathfrak{p}
\mathclose{\raisebox{-1.8ex}{$\Biggr)$}}
\]

where the parentheses have been shifted down

you can raise it but that doesn't look nice:

\documentclass{article}
\usepackage{amssymb}
\usepackage{amsmath}
\begin{document}

\[
\varphi^{-1}
\left(\array{@{}c@{}}
  \displaystyle\bigcap\\
  \substack{\mathfrak{p} < B \\ p \text{ prime}}\endarray 
\mathfrak{p}\right)
\]

\end{document}