Increase size of square brackets

There's a workaround for the brackets size: use \bigtrut[t] in a cell of the first row and \bigstrut[b] in the last row. I added a simplication of the code for the first column, using the BAenum row counter defined by the package Last, I removed the unnecessary colon after the figure label, since there's no caption text.

\documentclass{article}
\usepackage{xcolor}
\usepackage{blkarray, bigstrut}
\usepackage{caption}

\usepackage{amsmath}

\begin{document}

\begin{figure}[htbp]\bigstrutjot=1ex \captionsetup{labelsep=none, skip=0pt}
    \[ \mathbf{\widehat R }=\begin{blockarray}{r*{5}{ >{\color{black}}c}}
    & A & B & C & D & E \\\noalign{\setcounter{BAenumi}{1}}
    \begin{block}{ >{\scriptstyle\theBAenumi}r!{\,}[ccccc]}
    & 5 & 3.09 & 4.90 & - & \mathbf{4.62}\bigstrut[t] \\
    & - & 2.89 & 4.69 & \mathbf{3.49} & 4.42 \\
    & 4.10 & 2.19 & - & \mathbf{2.78} & 3.71 \\
    & - & \mathbf{1.00} & 2.49 & 1.29 & 2.22 \\
    & 4.90 & - & \mathbf{4.79} & 3.58 & 4.51 \\
    & \mathbf{4.88} & 2.96 & - & 3.56 & 4.48 \\
    & 3.15 & \mathbf{1.23} & 3.03 & 1.82 & - \\
    & 4.84 & \mathbf{2.92} & 4.72 & - & 4.61 \\
    & \mathbf{4.84} & 2.92 & 4.72 & 3.51 & - \\
    & \mathbf{4.61} & - & 4.49 & 3.29 & 4.22 \bigstrut[b]\\
    \end{block}
    \end{blockarray} \]%
\caption{}
\end{figure}

\end{document} 

In my opinion, increasing the total height of the square brackets should be the least of your (typographic) priorities. I'd focus most (all?) of my energy on telling TeX to use the regular-bold rather than (default) bold-extended font weight for the numbers in the five data columns. I would also expend some energy on increasing the horizontal separation between the tall [ and ] symbols and the columns they enclose. In the code shown below, that's accomplished with the >{\,} and <{\,} particles.

Oh, and don't use a spacing environment inside a display-math setting. If you absolutely, positively must shrink the distance between rows, reset the parameter \arraystretch to a value less than 1. In the example shown below, I use a value of 0.5 -- which, in my opinion, is much too small. Do try something like 0.8 or 0.9 instead.

\documentclass{article}
\usepackage{blkarray}

\begin{document}
\[ 
\renewcommand{\bfdefault}{b}   % use bold, not bold-extended
\renewcommand\arraystretch{.5} % choose a suitable factor
\widehat{\mathbf{R}}= 
\begin{blockarray}{@{} r *{5}{c} }
      & A & B & C & D & E \\
    \begin{block}{@{} >{\scriptscriptstyle}r [>{\,}ccccc<{\,}] }
    1 & 5.00 & 3.09  & 4.90 & - & \textbf{4.62} \\
    2 & - & 2.89 & 4.69  & \textbf{3.49} & 4.42 \\
    3 & 4.10 & 2.19 & - & \textbf{2.78}  & 3.71 \\ 
    4 & - & \textbf{1.00} & 2.49 & 1.29  & 2.22 \\
    5 & 4.90 & - & \textbf{4.79} & 3.58  & 4.51 \\
    6 & \textbf{4.88} & 2.96 & - & 3.56  & 4.48 \\
    7 & 3.15 & \textbf{1.23} & 3.03 & 1.82 & -  \\
    8 & 4.84 & \textbf{2.92} & 4.72 & -  & 4.61 \\
    9 & \textbf{4.84} & 2.92 & 4.72 & 3.51 & -  \\ 
    10& \textbf{4.61} & - & 4.49 & 3.29  & 4.22 \\
    \end{block}
    \end{blockarray} 
\]
\end{document}

You could consider using kbordermatrix:

\documentclass{article}

\usepackage{kbordermatrix}

\begin{document}

\[ 
  \renewcommand{\bfdefault}{b}   % use bold, not bold-extended
  \widehat{\mathbf{R}} = 
  \kbordermatrix{%
       &   A  &   B  &   C  & D & E \\
    1  & 5.00 & 3.09 & 4.90 & - & \textbf{4.62}  \\
    2  & - & 2.89 & 4.69  & \textbf{3.49} & 4.42 \\
    3  & 4.10 & 2.19 & - & \textbf{2.78}  & 3.71 \\
    4  & - & \textbf{1.00} & 2.49 & 1.29  & 2.22 \\
    5  & 4.90 & - & \textbf{4.79} & 3.58  & 4.51 \\
    6  & \textbf{4.88} & 2.96 & - & 3.56  & 4.48 \\
    7  & 3.15 & \textbf{1.23} & 3.03 & 1.82 & -  \\
    8  & 4.84 & \textbf{2.92} & 4.72 & -  & 4.61 \\
    9  & \textbf{4.84} & 2.92 & 4.72 & 3.51 & -  \\
    10 & \textbf{4.61} & - & 4.49 & 3.29  & 4.22
  }
\]

\end{document}

The above code is taken partially from Mico's answer.