Ugly alignment (size of square root symbol) with Latin Modern at 12pt

You have (re)discovered an important weakness in the Latin Modern math fonts. If we look at omxlmex.fd we see

% This file belongs to the Latin Modern package. The work is released
% under the GUST Font License. See the MANIFEST-Latin-Modern.txt and
% README-Latin-Modern.txt files for the details. For the most recent version of
% this license see http://www.gust.org.pl/fonts/licenses/GUST-FONT-LICENSE.txt
% or http://tug.org/fonts/licenses/GUST-FONT-LICENSE.txt

\ProvidesFile{omxlmex.fd}[2009/10/30 v1.6 Font defs for Latin Modern]

\DeclareFontFamily{OMX}{lmex}{}
\DeclareFontShape{OMX}{lmex}{m}{n}{%
   <->sfixed*lmex10%
   }{}
\endinput
%%
%% End of file `omxlmex.fd'.

This means that the math extension font is loaded at a fixed size, 10pt, independently of the document font. Let's try with

\documentclass[10pt]{article}
\usepackage{lmodern}
\begin{document}
\[
\sqrt{\frac{m}{n}}+\sum\frac{m}{n}+\int\frac{m}{n}
\]
\end{document}

Let's change 10pt into 12pt (with 11pt the problem is the same, but it's somewhat less easy to see):

Do you see? The fraction grew bigger, the symbols didn't. It's more evident if we typeset the example at 17pt (with extarticle):

Solution

\documentclass[12pt]{article}
\usepackage{lmodern}

\DeclareFontShape{OMX}{cmex}{m}{n}{%
  <-7.5> cmex7
  <7.5-8.5> cmex8
  <8.5-9.5> cmex9
  <9.5-> cmex10
}{}
\DeclareSymbolFont{largesymbols}{OMX}{cmex}{m}{n}

\begin{document}
\[
\sqrt{\frac{m}{n}}+\sum\frac{m}{n}+\int\frac{m}{n}
\]
\end{document}

And ask the Latin Modern team to fix this embarrassing glitch. Maybe if they get tons of mail they'll reconsider it.

Update

One can do \usepackage{fixcmex}

\documentclass[12pt]{article}
\usepackage{lmodern}
\usepackage{fixcmex}

\begin{document}
\[
\sqrt{\frac{m}{n}}+\sum\frac{m}{n}+\int\frac{m}{n}
\]
\end{document}

To reproduce the issue you've encountered, it's crucial that the document font size be set to 12pt. Loading the lmodern and mathtools packages may be more of a side show.

I take it that you don't like the fact that the horizontal rule of the square root symbol is too close to the letter "m". To fix this issue, you could insert a \mathstrut directive in the first argument of \frac. If you have a lot of \frac terms, you may want to set up a dedicated macro called, say, \myfrac, that automatically inserts a \mathstrut in the numerator terms. This is what's done in the third expression of the following example.

\documentclass[12pt]{article}           
\usepackage{lmodern,mathtools}
\newcommand\myfrac[2]{\frac{#1\mathstrut}{#2}}    
\begin{document}
\[ 
\sqrt{\frac{m}{n}} 
\quad 
\sqrt{\frac{m\mathstrut}{n}} 
\quad 
\sqrt{\myfrac{m}{n}} 
\]    
\end{document}

---

Addendum: If you must use 12pt as the main document size, you may want to consider switching from pdfLaTeX to LuaLaTeX and the unicode-math package. Doing so will yield much better spacing between the numerator term and the horizontal portion of the square root symbols, whether or not the amsmath package (or its superset, the mathtools package) is loaded. (If you do choose to load mathtools, you should do so before loading unicode-math.)

% !TeX program = lualatex  % tested with MacTeX2017
\documentclass[12pt]{article} 
%\usepackage{mathtools} % optional         
\usepackage{unicode-math}   
\begin{document}
\[ 
\sqrt{\frac{m}{n}} \quad \sqrt{\frac{m\mathstrut}{n}} 
\]    
\end{document}

This modifies the font configuration for Latin Modern in a way which I think is analogous to the modifications amsmath.sty makes for Computer Modern. It is not exactly analogous because Computer Modern provides additional sizes of the maths extension font, whereas Latin Modern provides only 10pt. However, it does seem --- in absolutely minimal testing --- to work in a similar way.

Caveat emptor ...

\documentclass[12pt]{article}
\makeatletter
% \usepackage[cmex10]{amsmath}
\usepackage{amsmath}
\usepackage{lmodern}
% from omxlmex.fd
\DeclareFontFamily{OMX}{lmex}{}
% modified from amsmath.sty
\ifnum\cmex@opt=7 \relax
  \DeclareFontShape{OMX}{lmex}{m}{n}{%
    <-8>lmex10<8>lmex10<9>lmex10%
    <10><10.95><12><14.4><17.28><20.74><24.88>lmex10%
  }{}%
  \expandafter\let\csname OMX/cmex/m/n/10\endcsname\relax
\else
  \ifnum\cmex@opt=\z@ % need to override cmex7 fontdef from amsfonts
    \begingroup
    \fontencoding{OMX}\fontfamily{lmex}%
    \expandafter\let\csname OMX+cmex\endcsname\relax
    \try@load@fontshape
    \endgroup
    \expandafter\let\csname OMX/cmex/m/n/10\endcsname\relax
    \def\cmex@opt{10}%
  \fi
\fi
\makeatother
\begin{document}
  \[ \sqrt{\frac{m}{n}} \sqrt[3]{\frac{m}{n}} \]
\end{document}