How do I nest a command with an optional argument?

If you define your command with xparse, the problem is taken care of automatically; note that you can also easily insert the subscript only if actually specified.

\documentclass{article}
\usepackage{xparse}

\NewDocumentCommand{\nrml}{om}{%
  N\IfValueT{#1}{_{#1}}(#2)%
}

\begin{document}

$\nrml[\nrml{H}]{J}$

$\nrml[\nrml[K]{H}]{J}$

\end{document}

Enclose the outer optional argument in braces. The issue is in the way that optional arguments are digested by LaTeX (see Bug? LaTeX misparses nested optional arguments).

\documentclass{article}
\newcommand{\nrml}[2][]{N_{#1}(#2)}

\begin{document}
$\nrml[\nrml{H}]{J}$

$\nrml[{\nrml[G]{H}}]{J}$
\end{document}