How to make frame in matrix?

A TikZ-free possibility:

\documentclass[11pt,twoside,openright]{book}
\usepackage{amsmath}

\begin{document}

\[
J_5=
\left(
\begin{array}{|cc|c@{}c|} 
\cline{1-2}
\lambda & 1 & & \multicolumn{1}{c}{0} \\
0 & \multicolumn{1}{c|}{\lambda} & & \multicolumn{1}{c}{0} \\
\cline{1-2}\cline{4-4}
\multicolumn{1}{c}{0} & \multicolumn{1}{c}{0} & & \multicolumn{1}{|c|}{\lambda} \\
\cline{4-4}
\end{array}
\right)
\]

\end{document}

Since you are loading tikz, I assume a tikz solution is fine.

It makes use of two tikz libraries: matrix for the matrix (the syntax is very close to the usual syntax for tabulars, arrays or matrices, with a lot of optional parameters to tweak the final output), and fit to draw rectangles around sets of nodes.

\documentclass[11pt,twoside,openright]{book}
\usepackage{geometry,tikz,amsmath}

\usetikzlibrary{fit,matrix}

\begin{document}

\begin{equation*}
  J_{5}=
  \tikz[baseline=(M.west)]{%
    \node[matrix of math nodes,matrix anchor=west,left delimiter=(,right delimiter=),ampersand replacement=\&] (M) {%
      \lambda \& 1 \& 0 \\
      0 \& \lambda \& 0 \\
      0 \& 0 \& \lambda \\
    };
    \node[draw,fit=(M-1-1)(M-2-2),inner sep=-1pt] {};
    \node[draw,fit=(M-3-3),inner sep=-1pt] {};
  }
\end{equation*}

\end{document}