Problems using `\left\{` in array environment

I took @mico's answer and added hhline

\documentclass[a4paper,12pt]{report}
\usepackage{multirow,hhline}
\begin{document}
  \[
  A =
  \left(
  \begin{array}{r|rrr||r|l}
    \hhline{~|---||-|~}
    \multirow{2}{*}{$B \bigg\{$} 
    & 1 & 2 & -4 & 1 & 
    \multirow{2}{*}{$\bigg\} \hat{\mathbf{b}}^{T}$} \\
    & 2 & -5 & -3 & -3 & \\
    \hhline{~:===::=:~}
    \hat{\mathbf{c}} \{  & -6 & -3 & 0 & 5 & \} a_{n-1,n} \\
    \hhline{~:===::=:~}
    \hat{\mathbf{d}} \{ & -6 &  7 & 0 & -8 & \} a_{n,n} \\
    \hhline{~|---||-|~}
  \end{array}
  \right)
  \]
\end{document}

Here's a solution using the tried and tested \tikzmark idea, first show cased in Andrew Stacey's answer to

Adding a large brace next to a body of text

The code below uses a version modified from Peter Grill's answer to

Box around a few items in an itemize environment

My solution is a little bit fiddly- there's a bit of manual column and row spacing, and a couple of uses of \vphantom so that instead of using \bigg you can use \left\{...\right. but the output is (I hope) quite pleasing.

\documentclass[a4,12pt]{report}

\usepackage{tikz}
\usetikzlibrary{calc}
\usepackage{multirow}

\newcommand{\DrawBox}[3][]{%
    \tikz[overlay,remember picture]{
        \coordinate (RightPoint) at (#3.east);
    \draw[red,#1]
      ($(#2)+(-.5em,.9em)$) rectangle
      ($(RightPoint)+(0.2em,-0.3em)$);}
}

\newcommand{\tikzmark}[1]{\tikz[overlay,remember picture] \node (#1) {};}

\begin{document}

\[
  A =
    \left(
    \begin{array}{r@{\hspace{.2cm}}c@{}rrr@{\hspace{.5cm}}r@{}rl}
       \multirow{2}{*}{$B \left\{\vphantom{\begin{array}{c}1\\2\end{array}}\right.$} 
                                &\tikzmark{left1}   & 1   & 2  & -4                     &\tikzmark{left3}   & 1  & 
                                \multirow{2}{*}{$\left.\vphantom{\begin{array}{c}1\\2\end{array}}\right\} \hat{\mathbf{b}}^{T}$} \\
                                &                   & 2   & -5 & -3\tikzmark{right1}    &                   &-3 \tikzmark{right3}   & \\[5pt]
        \hat{\mathbf{c}} \{     &\tikzmark{left2}   & -6  & -3 & 0 \tikzmark{right2}    &\tikzmark{left4}   & 5  \tikzmark{right4}  &\}a_{n-1,n} \\[5pt]
        \hat{\mathbf{d}} \{     &\tikzmark{left5}   & -6  &  7 & 0 \tikzmark{right5}    &\tikzmark{left6}   & -8 \tikzmark{right6}  & \} a_{n,n} \\
      \end{array}
    \right)
\]

\DrawBox[thick,green,fill=yellow,fill opacity=0.1]{left1}{right1}
\DrawBox[thick,blue]{left2}{right2}
\DrawBox[thick,red]{left3}{right3}
\DrawBox[thick,purple]{left4}{right4}
\DrawBox[thick,yellow]{left5}{right5}
\DrawBox[thick,orange]{left6}{right6}

\end{document}

You could use the explicit size statement \bigg in order to set the size of the left and right curly braces, as is shown in the following modified form of your MWE. To get the arrays display with a little gap while using the \cline command, it's necessary to break them up into two separteI've also simplified the code to get rid of most of the \multicolumn statements and combining the various paired \cline statements.

\documentclass[a4paper,12pt]{report}
\usepackage{multirow}
\begin{document}      
  \[
  A =
  \left(
  \begin{array}{r|rrr|}
    \cline{2-4} 
    \multirow{2}{*}{$B \bigg\{$} 
    & 1 &  2 & -4  \\ 
    & 2 & -5 & -3 \\
    \cline{2-4} 
    \hat{\mathbf{c}} \{  & -6 & -3 & 0 \\
    \cline{2-4}
    \hat{\mathbf{d}} \{  & -6 &  7 & 0 \\
    \cline{2-4} 
  \end{array}
  \right. \kern-3pt \left. % use \kern-3pt to reduce distance between arrays
  \begin{array}{|r|l}
    \cline{1-1}
     1 & \multirow{2}{*}{$\bigg\} \hat{\mathbf{b}}^{T}$} \\
    -3 & \\
    \cline{1-1}
     5 & \} a_{n-1,n} \\
    \cline{1-1}
    -8 & \} a_{n,n} \vphantom{\hat{\mathbf{d}}}\\  
        % the \vphantom is needed to equalize the heights of the 
        % third rows across the two arrays
    \cline{1-1}
  \end{array}
  \right)
  \]
\end{document}