Strange behaviour of binomial coefficient's delimiters

This is a problem in how lmodern sets up the math extensions font, by saying

\DeclareFontShape{OMX}{lmex}{m}{n}{%
   <->sfixed*lmex10%
   }{}

which is utterly wrong, in my opinion, because it destroys the settings made by amsmath which would solve the problem. The fonts, unfortunately, do not provide for optical sizes different from 10pt, so you have a few strategies

1. Use the fixcmex package

\documentclass[11pt]{article}

\usepackage{amsmath}
\usepackage{lmodern}
\usepackage{fixcmex}

\usepackage{lipsum}

\begin{document}

\lipsum[1]
\begin{equation}
2^k-\binom{k}{1}2^{k-1}+\binom{k}{2}2^{k-2}
\end{equation}
\lipsum[1]

\end{document}

Disclaimer. I authored the fixcmex package as a solution to the present problem.

2. Let amsmath do the main work

\documentclass[11pt]{article}

\usepackage{amsmath}
\usepackage{lmodern}

\DeclareSymbolFont{largesymbols}{OMX}{cmex}{m}{n} % use cmex rather than lmex

\usepackage{lipsum}

\begin{document}

\lipsum[1]
\begin{equation}
2^k-\binom{k}{1}2^{k-1}+\binom{k}{2}2^{k-2}
\end{equation}
\lipsum[1]

\end{document}

3. Load exscale or, better, the fonts defined by it

\documentclass[11pt]{article}

\usepackage{amsmath}
\usepackage{lmodern}

\DeclareFontShape{OMX}{cmex}{m}{n}{%
      <-7.5> cmex7
      <7.5-8.5> cmex8
      <8.5-9.5> cmex9
      <9.5-> cmex10
       }{}
\DeclareSymbolFont{largesymbols}{OMX}{cmex}{m}{n} % use cmex rather than lmex

\usepackage{lipsum}

\begin{document}

\lipsum[1]
\begin{equation}
2^k-\binom{k}{1}2^{k-1}+\binom{k}{2}2^{k-2}
\end{equation}
\lipsum[1]

\end{document}

4. Make lmex10 scalable

\documentclass[11pt]{article}

\usepackage{amsmath}
\usepackage{lmodern}
\DeclareFontFamily{OMX}{lmex}{}
\DeclareFontShape{OMX}{lmex}{m}{n}{%
   <-> lmex10
   }{}

\usepackage{lipsum}

\begin{document}

\lipsum[1]
\begin{equation}
2^k-\binom{k}{1}2^{k-1}+\binom{k}{2}2^{k-2}
\end{equation}
\lipsum[1]

\end{document}

With all strategies the computation for the delimiter's sizes gives a better result. Strategies two and three are rather similar; the difference is that the latter scales all fonts even at non standard sizes, while the former only uses fixed sizes 7, 8, 9, 10, 11 (10.95), 12, 14 (14.4) and 17 (17.28). Strategy one is equivalent to strategy three (and easier to load).

This answer relies on redefining \binom to use features of the scalerel and stackengine packages. The \scaleleftright macro will make the paren delimiters exactly match the height of the binomial contents, which are stacked using \stackanchor.

The vertical gap between the components of the binomial coefficient is an optional argument to \stackanchor (currently set at 1.8ex), and the horizontal limiting width of the parens of the coefficient are given as an optional argument to \scaleleftright (currently set to 1.5ex).

\documentclass[11pt]{article}
%
\usepackage{amsmath}
\usepackage{scalerel}
\usepackage{stackengine}
%
\makeatletter
%
\usepackage{lmodern}
\usepackage{lipsum}
%
\makeatother
%
\begin{document}
%
\lipsum[1]
\begin{equation}
2^k-\binom{k}{1}2^{k-1}+\binom{k}{2}2^{k-2}
\end{equation}
\renewcommand\binom[2]{\stackMath\mathop{%
  \scaleleftright[1.5ex]{(}{\stackanchor[1.8ex]{#1}{#2}}{)}}}
\begin{equation}
2^k-\binom{k}{1}2^{k-1}+\binom{k}{2}2^{k-2}
\end{equation}
\lipsum[1]

%
\end{document}

The top image is the original that you posted, while the lower one is the fix proposed here.