How do I label different rows or columns of a matrix using braces?
The following solution uses TikZ, but a minimal amount of hacks:
\usepackage{tikz}
\usetikzlibrary{matrix,decorations.pathreplacing}
\begin{tikzpicture}[decoration=brace]
\matrix (m) [matrix of math nodes,left delimiter=[,right delimiter={]}] {
a & b & c & d \\
e & f & g & h \\
i & j & k & l \\
m & n & o & p \\
};
\draw[decorate,transform canvas={xshift=-1.5em},thick] (m-3-1.south west) -- node[left=2pt] {$A$} (m-1-1.north west);
\draw[decorate,transform canvas={yshift=0.5em},thick] (m-1-2.north west) -- node[above=2pt] {$R$} (m-1-4.north east);
\end{tikzpicture}
First two tikz libraries are loaded (this only needs to be done once). In the tikzpicture environment a matrix called m is defined. The delimiters can be changed to anything that is acceptable after \left in math mode. Entries of the matrix can be reference by name-row-column (where in this case the name is m). I found good looking shifts (i.e. the distances between brace and matrix entry) by trail-and-error; you might have to change them. Also you might want to try removing the thick parameter.
I tried wrapping the whole stuff in nice commands. Unfortunately I only succeeded partially, because I do not know how to handle the & in the matrix without LaTeX complaining about "misplaced alignment tab character". I tweaked the spacing in the following tikz styles a bit to make the matrix look more like the usual ones. Maybe there are better ways to do this (are all the lengths the (usual) matrix environment used accessible somehow?).
Anyway, on to the commands. Put the following wall of code somewhere near the beginning of the document:
% Load TikZ
\usepackage{tikz}
\usetikzlibrary{matrix,decorations.pathreplacing,calc}
% Set various styles for the matrices and braces. It might pay off to fiddle around with the values a little bit
\pgfkeys{tikz/mymatrixenv/.style={decoration=brace,every left delimiter/.style={xshift=3pt},every right delimiter/.style={xshift=-3pt}}}
\pgfkeys{tikz/mymatrix/.style={matrix of math nodes,left delimiter=[,right delimiter={]},inner sep=2pt,column sep=1em,row sep=0.5em,nodes={inner sep=0pt}}}
\pgfkeys{tikz/mymatrixbrace/.style={decorate,thick}}
\newcommand\mymatrixbraceoffseth{0.5em}
\newcommand\mymatrixbraceoffsetv{0.2em}
% Now the commands to produce the braces. (I'll explain below how to use them.)
\newcommand*\mymatrixbraceright[4][m]{
\draw[mymatrixbrace] ($(#1.north west)!(#1-#3-1.south west)!(#1.south west)-(\mymatrixbraceoffseth,0)$)
-- node[left=2pt] {#4}
($(#1.north west)!(#1-#2-1.north west)!(#1.south west)-(\mymatrixbraceoffseth,0)$);
}
\newcommand*\mymatrixbraceleft[4][m]{
\draw[mymatrixbrace] ($(#1.north east)!(#1-#2-1.north east)!(#1.south east)+(\mymatrixbraceoffseth,0)$)
-- node[right=2pt] {#4}
($(#1.north east)!(#1-#3-1.south east)!(#1.south east)+(\mymatrixbraceoffseth,0)$);
}
\newcommand*\mymatrixbracetop[4][m]{
\draw[mymatrixbrace] ($(#1.north west)!(#1-1-#2.north west)!(#1.north east)+(0,\mymatrixbraceoffsetv)$)
-- node[above=2pt] {#4}
($(#1.north west)!(#1-1-#3.north east)!(#1.north east)+(0,\mymatrixbraceoffsetv)$);
}
\newcommand*\mymatrixbracebottom[4][m]{
\draw[mymatrixbrace] ($(#1.south west)!(#1-1-#3.south east)!(#1.south east)-(0,\mymatrixbraceoffsetv)$)
-- node[below=2pt] {#4}
($(#1.south west)!(#1-1-#2.south west)!(#1.south east)-(0,\mymatrixbraceoffsetv)$);
}
After that, you can simply use the following code to produce your example:
\[
\begin{tikzpicture}[mymatrixenv]
\matrix[mymatrix] (m) {
a & b & c & d \\
e & f & g & h \\
i & j & k & l \\
m & n & o & p \\
};
\mymatrixbraceright{1}{3}{$A$}
\mymatrixbracetop{2}{4}{$R$}
\end{tikzpicture}
\]
The matrix definition remained pretty much the same as before, except that all the styles needed are now stored in mymatrixenv and mymatrix. No need to repeat them for every matrix!
The braces are produced with \mymatrixbraceright, \mymatrixbracetop, \mymatrixbraceleft and \mymatrixbracebottom for right, top, left and bottom braces respectively. The commands all work the same way: They have three mandatory commands for starting and ending position of the brace (in "natural" order) and the label. Additionally they have one optional argument for specifying the name of the matrix (it defaults to m). So, if you want a brace on the left of matrix foo going from the row 3 to row 417 and with label $415$ rows!, you use
\mymatrixbraceleft[foo]{3}{417}{$415$ rows!}
edit in 2017 to fix the site bug affecting \\'s
Here is one possible way to do it. It uses the mathtools package for its \mathrlap macro. My earlier answer used also the \rotatebox of the graphicx package but after reading
Align large vertical delimiters (brackets) to rows of a table?
I realized how silly I had been for the vertical braces. So I have completely replaced my code with this version (which does not at all need graphicx anymore); it does need mathtools because I have typeset a slightly complicated case, with simpler situations one could make do without \mathrlap.
I have included an image of the result. The code pays attention that the inked area ends up entirely enclosed in its bounding box, so that it does not print over material either to the left, the right, the top or the bottom.
It's a bit hard to write a single macro to automatize it, as the mark-up is best left to the clever representatives of the homo sapiens sapiens genus. It is pretty much self-explanatory how to adapt it to any given situation.
Defects: the vertical braces extend a bit far compared to the horizontal ones. But I don't see how to fix this without playing with some of the dimensions used internally by the matrix environment. So I left the matter here.
\documentclass{article}
\usepackage{amsmath}
\usepackage{mathtools}
\pagestyle{empty}
\newcommand\coolover[2]{\mathrlap{\smash{\overbrace{\phantom{%
\begin{matrix} #2 \end{matrix}}}^{\mbox{$#1$}}}}#2}
\newcommand\coolunder[2]{\mathrlap{\smash{\underbrace{\phantom{%
\begin{matrix} #2 \end{matrix}}}_{\mbox{$#1$}}}}#2}
\newcommand\coolleftbrace[2]{%
#1\left\{\vphantom{\begin{matrix} #2 \end{matrix}}\right.}
\newcommand\coolrightbrace[2]{%
\left.\vphantom{\begin{matrix} #1 \end{matrix}}\right\}#2}
\begin{document}
\[ \vphantom{% phantom stuff for correct box dimensions
\begin{matrix}
\overbrace{XYZ}^{\mbox{$R$}}\\ \\ \\ \\ \\ \\
\underbrace{pqr}_{\mbox{$S$}}
\end{matrix}}%
\begin{matrix}% matrix for left braces
\vphantom{a}\\
\coolleftbrace{A}{e \\ y\\ y}\\
\coolleftbrace{B}{y \\i \\ m}
\end{matrix}%
\begin{bmatrix}
a & \coolover{R}{b & c & d} & x & \coolover{Z}{x & x}\\
e & f & g & h & x & x & x \\
y & y & y & y & y & y & y \\
y & y & y & y & y & y & y \\
y & y & y & y & y & y & y \\
i & j & k & l & x & x & x \\
m & \coolunder{S}{n & o} & \coolunder{W}{p & x & x} & x
\end{bmatrix}%
\begin{matrix}% matrix for right braces
\coolrightbrace{x \\ x \\ y\\ y}{T}\\
\coolrightbrace{y \\ y \\ x }{U}
\end{matrix}\]
\end{document}
This solution is not as elegant as a dedicated package would (hopefully) be, but I don't know of one so here goes... This example assumes that you are using the amsmath package.
You can achieve the desired effect by creating a 2x2 'super matrix' and leaving the (1,1) entry blank. Then put a horizontal brace spanning some columns in the (1,2) entry and a vertical brace spanning some rows in the (2,1) entry. Finally, put the actual matrix that these braces are spanning in the (2,2) entry.
To get the spacing right, you can use \vphantom and \hphantom to create invisible matrices in the (1,2) and (2,1) entries, and the \mspace command to decrease the spacing between the two columns of the super matrix..
Here's an example (that may look much better pasted into an editor that has syntax highlighting):
\begin{matrix} %This is the super matrix
%(1,1) cell: Empty
&
%(1,2) cell: Brace spanning columns
\begin{matrix} %One-row matrix to hold the brace
\hphantom{a} & %Blank space to skip over first column
\overbrace{
\hphantom{\begin{matrix}b & c & d\end{matrix}}
}^{\text{\footnotesize Final three columns}}
\end{matrix}
\\
%(2,1) cell: Brace spanning rows
\begin{matrix} %One-column matrix to hold a brace
\vphantom{a} \\ %Blank space to skip first row
\text{\footnotesize Middle two rows}
\left\{\vphantom{\begin{matrix} e \\ i \end{matrix}}\right. \\
\vphantom{m}
\end{matrix}
%The inter-column spacing of the super matrix looks too big by default
\mspace{-33mu}
&
%(2,2) cell: Actual matrix
\begin{pmatrix}
a & b & c & d \\
e & f & g & h \\
i & j & k & l \\
m & n & o & p
\end{pmatrix}
\end{matrix}
The invisible matrices in the (1,2) and (2,1) entries of the super matrix allow you to span whichever columns you want, or even have multiple braces in either direction.