How do I create a bordered Hessian matrix in LaTeX?

Here is how I would do it:

\documentclass{article}

\usepackage{mathtools}
\usepackage{bm}

\DeclareMathOperator{\Hess}{Hess}
\newcommand*\vect[1]{\mathbf{#1}}

% http://tex.stackexchange.com/a/59955/15874
% The following piece of code measures the length of math expressions
% in order to get the length of the two horizontal rules (based on
% the chosen width of the whitespaces) in the matrix.
\makeatletter
\newcommand*\settowidthofalign[2]{%
  \setbox\z@=\vbox{
    \begin{align*}
    #2
    \ifmeasuring@\else\global\let\got@maxcolwd\maxcolumn@widths\fi
    \end{align*}
  }%
  \begingroup
  \def\or{+}\edef\x{\endgroup#1=\dimexpr\got@maxcolwd\relax}\x}
\makeatother

\newlength{\mylenA}
\newlength{\mylenB}

% Calculate length of rules.
\if \mylenA < \mylenB
  \newcommand*\ruleLength{\the\dimexpr(\mylenA-\mylenB-4*(\WhiteSpace))/2\relax}
 \else
  \newcommand*\ruleLength{\the\dimexpr(\mylenB-\mylenA-4*(\WhiteSpace))/2\relax}
\fi

% Matrix construction.
\newcommand*\Matrix[4]{%
  \left[
    \begin{array}{@{} c | c @{}}
      #1
      & \begin{array}{c}
          \vert\\
          #2\\
          \vline
          \vspace{0.5ex}
         \end{array}\\
      \hline
      \vspace{0.5ex}
      \raisebox{-0.3ex}{%
        $\rule[0.5ex]{\ruleLength}{0.4pt}
         \hspace{\WhiteSpace} #3 \hspace{\WhiteSpace}
         \rule[0.5ex]{\ruleLength}{0.4pt}$%
      }
      & \raisebox{-0.3ex}{$#4$}
    \end{array}
  \right]%
}

% Matrix entries.
\newcommand*\entryA{\Hess(f)(\vect{a}) - \lambda\Hess(g)(\vect{a})}
\newcommand*\entryB{\bm{\nabla} g(\vect{a})}
\newcommand*\entryC{D g(\vect{a})}
\newcommand*\entryD{0}

\begin{document}

% Widths of expressions in matrix entries.
\settowidthofalign{\mylenA}{\entryA}
\settowidthofalign{\mylenB}{\entryB}

\begin{equation*}
  % Width of whitespace on each side of horizontal lines.
  \def\WhiteSpace{12pt}
  % Math expression.
  H_{f,g} \binom{\vect{a}}{\lambda}
  = \Matrix{\entryA}{\entryB}{\entryC}{\entryD}
\end{equation*}

\end{document}

This might be overkill in this particular case but you now have a working construction that can be re-used for other matrices. All you have to do is

• define the entries of the new matrix (e.g., \entryE),
• define the new lengths to be used (using \newlength),
• let TeX calculate the widths of the math expressions (using \settowidthofalign),
• typeset the whole shebang (using \Matrix).

Note: I wouldn't make the \nabla bold; I think it disturbes the reading more than it helps.

Here is an approach with an array and some \rule commands for the extra lines inside of cells. Note that the array actually consists of four rows.

Code

\documentclass[varwidth]{standalone}
\usepackage{amsmath,bm}
\DeclareMathOperator\Hess{Hess}
\begin{document}
\begin{equation*}
\renewcommand*\a{\mathbf{a}}
\newcommand*\vl{\rule[-.3em]{.4pt}{1em}}
\newcommand*\hl{\rule[.5ex]{2em}{.4pt}}
%
H_{f,g}\binom{\a}{\lambda} = \left[\begin{array}{c|c}
                                  & \vl          \\
 \Hess(f)(\a)-\lambda\Hess(g)(\a) & \nabla g(\a) \\
                                  & \vl          \\ \hline
 \hl\quad Dg(\a) \quad\hl         & 0
\end{array}\right]
\end{equation*}
\end{document}

Output

Something like that?

\documentclass{article}
\usepackage{amssymb}
\usepackage{mathtools, array}
\DeclareMathOperator{\hess}{Hess}

\def\longline{\relbar\mkern-6mu\relbar}

\begin{document}

\[\setlength\extrarowheight{1pt}
  \begin{bmatrix}
    \begin{array}{c@{\,}c@{\,}c|c}
    & & & \vrule \\
    \multicolumn{3}{c|}{\hess_{f, g}(f)(\mathbf a)} & \nabla g(\mathbf a)\\
      & & & \vrule \\[-2.6ex] & & & \\
    \hline
    \longline & Dg(\mathbf a) &\longline & 0
    \end{array}
  \end{bmatrix} \]%

\end{document}