Matrix in TikZ and Equation

You're nesting tikzpictures, that is often a bad idea. Here's an alternative approach using the fit library.

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{matrix,decorations.pathreplacing, calc, positioning,fit}
\begin{document}
\begin{equation}
  Q= 
\begin{tikzpicture}[
  baseline,
  label distance=10pt % added
]


\matrix [matrix of math nodes,left delimiter=(,right delimiter=),row sep=0.1cm,column sep=0.1cm] (m) {
      1 & *     &   *  & \dots & *   & 0 & \dots & 0 \\
      0 & \pm 1 &   *  & \dots & *   & 0 & \dots & 0 \\
      0 & 0     &\pm 1 & \dots & *   & 0 & \dots & 0 \\
      \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \ddots & \vdots\\
        0 &  0    & 0    &   \dots   &\pm 1& 0 & \dots & 0 \\
        0 &  0    & 0    &  \dots   & 0  & * & \dots & * \\
      \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \ddots & \vdots\\
      0 &  0    & 0    &   0   & 0   & * & \dots & *\\ };

\draw[dashed] ($0.5*(m-1-5.north east)+0.5*(m-1-6.north west)$) -- ($0.5*(m-8-6.south east)+0.5*(m-8-5.south west)$);

\draw[dashed] ($0.5*(m-5-1.south west)+0.5*(m-6-1.north west)$) -- ($0.5*(m-5-8.south east)+0.5*(m-6-8.north east)$);

\node[
  fit=(m-1-1)(m-1-5),
  inner xsep=0,
  above delimiter=\{,
  label=above:$k_1$
] {};

\node[
  fit=(m-1-6)(m-1-8),
  inner xsep=0,
  above delimiter=\{,
  label=above:$k-k_1$
 ] {};

\node[
  fit=(m-1-8)(m-5-8),
  inner xsep=15pt,inner ysep=0,
  right delimiter=\},
  label=right:$k_1$
] {};

\node[
  fit=(m-6-8)(m-8-8),
  inner xsep=15pt,inner ysep=0,
  right delimiter=\},
  label=right:$k-k_1$
] {};

\end{tikzpicture} 
\end{equation}

\end{document}

Since you are loading already the decorations.pathreplacing library and make all the preparations, I could not resist using them. ;-)

\documentclass{article} 
\usepackage{tikz} 
\usetikzlibrary{matrix,decorations.pathreplacing, calc, positioning} 
\begin{document} 
\begin{equation}  Q= 
\begin{tikzpicture}[baseline] 
\matrix [matrix of math nodes,left delimiter=(,right delimiter=),row sep=0.1cm,column sep=0.1cm] (m) {
      1 & *     &   *  & \dots & *   & 0 & \dots & 0 \\
      0 & \pm 1 &   *  & \dots & *   & 0 & \dots & 0 \\
      0 & 0     &\pm 1 & \dots & *   & 0 & \dots & 0 \\
      \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \ddots & \vdots\\
        0 &  0    & 0    &   \dots   &\pm 1& 0 & \dots & 0 \\
        0 &  0    & 0    &  \dots   & 0  & * & \dots & * \\
      \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \ddots & \vdots\\
      0 &  0    & 0    &   0   & 0   & * & \dots & *\\ };

\draw[dashed] ($0.5*(m-1-5.north east)+0.5*(m-1-6.north west)$) -- ($0.5*(m-8-6.south east)+0.5*(m-8-5.south west)$);

\draw[dashed] ($0.5*(m-5-1.south west)+0.5*(m-6-1.north west)$) -- ($0.5*(m-5-8.south east)+0.5*(m-6-8.north east)$);

\node[above=1pt of m-1-1] (top-1) {}; 
\node[above=1pt of m-1-5] (top-5) {}; 
\node[above=-1pt of m-1-6] (top-6) {}; 
\node[above=-1pt of m-1-8] (top-8) {};

\node[right=10pt of m-1-8] (right-1) {}; 
\node[right=10pt of m-5-8] (right-5) {}; 
\node[right=10pt of m-6-8] (right-6) {}; 
\node[right=10pt of m-8-8] (right-8) {};

\draw[thick,decorate,decoration=brace] (top-1.west) -- (top-5.east)
node[midway,above=1mm] {$k_1$};

\draw[thick,decorate,decoration=brace] (top-6.west) -- (top-8.east)
node[midway,above=1mm] {$k-k_1$};

\draw[thick,decorate,decoration=brace] (right-1.north) -- (right-5.south)
node[midway,right=1mm] {$k_1$};

\draw[thick,decorate,decoration=brace] (right-6.north) -- (right-8.south)
node[midway,right=1mm] {$k-k_1$};

\end{tikzpicture} 
\end{equation}

\end{document}