\norm symbol have different sizes in equation

This is because the subscripts i and j have different depths and you are using automatic scaling.

There are two easy ways to fix this, since you are already using the mathtools and commath packages:

Solution with mathtools (preferred solution^†)

Make the \norm symbols the same size by using \DeclarePairedDelimiter from mathtools to declare \norm, and give a size as an optional argument as \norm[\big]{...}. Since you are using the commath package, which itself defines \norm, you can "save" and "undefine" the old definition of \norm before (re)declaring it using mathtools. You should also place the prime outside of \mathbf{...}, otherwise the subscript is placed too far right.

\documentclass{book}
\usepackage{amsmath,amsfonts,amssymb,amsthm}
\usepackage{mathtools}
\usepackage{commath}
\usepackage[sc,osf]{mathpazo}

\let\oldnorm\norm   % <-- Store original \norm as \oldnorm
\let\norm\undefined % <-- "Undefine" \norm
\DeclarePairedDelimiter\norm{\lVert}{\rVert}

\begin{document}

\begin{equation*}
  \mu(\mathbf{A}) = \max_{1\leqslant i,j\leqslant n, i\neq j}\dfrac{|\mathbf{A}'_{:,i}\mathbf{A}_{:,j}|}{\norm[\big]{\mathbf{A}_{:,i}}\norm[\big]{\mathbf{A}_{:,j}}}
\end{equation*}

\end{document}

or without optional argument (most correct size in my opinion), i.e. with {\norm{\mathbf{A}_{:,i}}\norm{\mathbf{A}_{:,j}}:

Solution with commath

The \norm command from the commath also takes an optional argument, ranging from 0 to 4 where 0 is the smallest size and 4 the biggest. Notice how the spacing before the first \norm is wrong – you will have to manually adjust this with e.g. \; to get a pleasing result.

\documentclass{book}
\usepackage{amsmath,amsfonts,amssymb,amsthm}
\usepackage{mathtools}
\usepackage{commath}
\usepackage[sc,osf]{mathpazo}

\begin{document}

\begin{equation*}
  \mu(\mathbf{A}) = \max_{1\leqslant i,j\leqslant n, i\neq j}\dfrac{|\mathbf{A}'_{:,i}\mathbf{A}_{:,j}|}{\norm[1]{\mathbf{A}_{:,i}}\norm[1]{\mathbf{A}_{:,j}}}
\end{equation*}

\end{document}

---

You should consider using \abs instead of |...| as well. Same approach as above with mathtools:

\let\oldabs\abs % Store original \abs as \oldabs
\let\abs\undefined % "Undefine" \abs
\DeclarePairedDelimiter\abs{\lvert}{\rvert}

and then use \abs*{...} to get automatic sizing, or with optional argument to manually size \abs[\big]{...}.

For commath the approach is also the same as above: \abs[1]{...}.

---

^†As you can see, the spacing before the first \norm in the denominator with commath (last image) is wrong, so personally I'd recommend using the mathtools approach, as that gives the correct spacing.

A prime example why \left and \right should not be used without checking. I can't recommend using commath macros for several reasons, see https://tex.stackexchange.com/search?q=commath+user%3A4427 for some of them.

The problem of using \left and \right shows here in full clarity: the descender in j is the cause of the sudden growth of the delimiters' size.

Let's simplify the situation. Look at the following code

\documentclass{article}
\usepackage[osf,sc]{mathpazo}
\usepackage{amsmath}

\begin{document}

\begin{gather}
\left\|\mathbf{A}_{i}\right\|\ne
\left\|\mathbf{A}'_{i}\right\|\ne
\left\|\mathbf{A}_{j}\right\|\ne
\left\|\mathbf{A}'_{j}\right\|
\\
\left\|\mathbf{A}_{i}\right\|+\left\|\mathbf{A}_{j}\right\|
\\
\lVert\mathbf{A}_{i}\rVert+\lVert\mathbf{A}_{j}\rVert
\\
\left\|\mathbf{A}'_{i}\right\|+\left\|\mathbf{A}'_{j}\right\|
\\
\lVert\mathbf{A}'_{i}\rVert+\lVert\mathbf{A}'_{j}\rVert
\end{gather}

\end{document}

and the output

Using \left\| and \right\| is basically the same as using \norm from commath.

You can notice that there are three delimiter sizes in line 1, two in lines 2 and 4. In particular, when \mathbf{A}'_{j} is used, the size is definitely too big.

It's clear to me that the “right” output is that in lines 3 and 5, with no size increase.

If you use mathtools, you can define a correct \norm macro and do manual adjustment only when necessary, which is not the case of your formula. I also suggest \substack for the long subscript.

\documentclass{article}
\usepackage[osf,sc]{mathpazo}
\usepackage{amsmath,mathtools,amssymb}

\DeclarePairedDelimiter{\norm}{\lVert}{\rVert}

\begin{document}

\begin{equation*}
\mu(\mathbf{A}) =
\max_{1\leqslant i,j\leqslant n, i\neq j}
  \dfrac{|\mathbf{A'}_{:,i}\mathbf{A}_{:,j}|}
        {\norm{\mathbf{A}_{:,i}}\norm{\mathbf{A}_{:,j}}}
\end{equation*}

\begin{equation*}
\mu(\mathbf{A}) =
\max_{\substack{1\leqslant i,j\leqslant n \\ i\neq j}}
  \dfrac{|\mathbf{A'}_{:,i}\mathbf{A}_{:,j}|}
        {\norm{\mathbf{A}_{:,i}}\norm{\mathbf{A}_{:,j}}}
\end{equation*}

\end{document}

The size of the first \norm command doesn't correspond to any of the predefined sizes which can be used as an optional argument. So the only solution I can imagine is to \smash j:

\documentclass{book}
\usepackage{amsmath,amsfonts,amssymb,amsthm}
\usepackage{mathtools}
\usepackage{commath}
\usepackage[sc,osf]{mathpazo}

\begin{document}

\begin{equation*}
  \mu(\mathbf{A}) = \max_{1\leqslant i,j\leqslant n, i\neq j}\dfrac{|\mathbf{A'}_{:,i}\mathbf{A}_{:,j}|}{\norm{\mathbf{A}_{:,i}}\norm{\mathbf{A}_{:,\smash[b]{j}}}}
\end{equation*}

\end{document}