Tex capacity exceeded compiling only 50 lines latex file (pdftex), seg. fault (lualatex)

The error is due to breqn, of course. Only one definition for an active character can be valid at a given time.

Since you have a math comma in the section title, this goes in the header and, with that long verbatim, it gets typeset when verbatim is active, which gives a different meaning to the active comma.

Here's a minimal example showing the issue.

\documentclass[11pt]{book}
\usepackage{breqn}

\setlength{\textheight}{2cm} % to shorten the input

\begin{document}
\section{$f(x,y)$}

\begin{verbatim}
x
x
x
x
x
\end{verbatim}
\end{document}

Here's a fix to \do@noligs, which is responsible for defining a new meaning for active characters in verbatim.

\documentclass[11pt]{book}
\usepackage{breqn}

\makeatletter
\def\do@noligs#1{%
  \catcode`#1\active
  \begingroup\lccode`\~`#1\relax
  \lowercase{\endgroup
  \ifdefined~%
    \expandafter\def\expandafter~\expandafter{%
      \expandafter\relax
      \expandafter\ifmmode
        ~% in math mode use the meaning given by breqn
      \else
        \leavevmode\kern\z@\char`#1\relax
      \fi
    }%
  \else
    \def~{\leavevmode\kern\z@\char`#1}%
  \fi
  }%
}
\makeatother

\setlength{\textheight}{2cm}

\begin{document}
\section{$f(x,y)$}

\begin{verbatim}
x
x
x
x
x
\end{verbatim}
\end{document}

The main problem is the breqn math comma being written to the mark used in headings while verbatim is active. which gives a classic fragile command in moving argument cryptic error. \hbox hides this (a better fix allowing the comma to be used normally may be posted later)

The fact that luatex segfaults after the runaway argument is a separate lower level bug (or not a bug). The code uses arbitrary amounts of memory as it is in an infinite loop, so in classic tex it hits a hard limit but luatex just allocates more and more memory until it runs out of memory of your actual hardware.

\documentclass[11pt]{book}%ext
\usepackage[T1]{fontenc}
\usepackage{verbatim}
\usepackage{amsmath,mathtools}
\usepackage[utf8]{inputenc} %DO NOT USE WIT LUALATEX, only with pdflatex
\usepackage{graphicx}
\usepackage{breqn}   %do not use this with tex4ht.  added 4/2/16



\begin{document}

\section{How to implement Steepest descent for finding minumum of  $f(x\hbox{,}y)$ ?}


\begin{verbatim}

    function nma_problem_2()
    close all;
    del      = 0.05;
    u1       = 0:del:20;
    u2       = -10:del:15;
    [u1,u2]  = meshgrid(u1,u2);

    plot(0,0);
    xlim([0 20]); ylim([-5 15]);
    hold on;
    [u(1),u(2)] = ginput(1);
    t=text(1.08*u(1),u(2),sprintf('[%2.1f,%2.1f]',u(1),u(2)));
    t.FontSize=8;
    t.Color='red';
    x = fminsearch(@objectiveFunc, u); %used to find the minumum in order to show
    %progress towards it
    hold on;
    plot(x(1),x(2),'*r')
    plot(u(1),u(2),'or')

    MAX_ITER = 50;
    REL_TOL  = 10^(-4);
    stepSize = -1; %optimal
    [p,levelSets] = steepestDescentPoints(u,MAX_ITER,REL_TOL,stepSize,...
    @objectiveFunc,@gradientFunc);
    [C,h]    = contour(u1,u2,z,levelSets); %,'ShowText','on');

    if length(p(:,1))>1
    for k=1:length(p)-1
    line([p(k,1),p(k+1,1)],[p(k,2),p(k+1,2)]); %draw line between each step
    drawnow;
    pause(.2);
    title(sprintf('starting from [%3.3f,%3.3f], f(u)=%3.3f,  step = %d',...
    u(1),u(2),norm(objectiveFunc(u)),k));
    end
    end
\end{verbatim}

\end{document}