Horizontal curly braces inside matrix
Here there is my idea: to have the curly braces inside the matrix of your determinant I have used always underbrace
command.
\documentclass[a4paper,12pt]{article}
\usepackage{amsmath,amssymb}
\begin{document}
\begin{equation}
R(f,g)=
\begin{vmatrix}
\underbrace{\begin{matrix}
a_0 & & & \\
a_1 & a_0 & & \\
a_2 & a_1 & a_0 & \\
& \ldots & & \\
& & & a_m
\end{matrix}}_{n \text{ columns}} & \underbrace{\begin{matrix}
b_0 & & & \\
b_1 & b_0 & & \\
b_2 & b_1 & b_0 & \\
& \ldots & & \\
& & & b_m
\end{matrix}}_{m \text{ columns}}\\
\end{vmatrix}
\end{equation}
\end{document}
This MWE have the curly brackets more inside:
\documentclass[a4paper,12pt]{article}
\usepackage{amsmath,amssymb}
\begin{document}
\begin{equation}
R(f,g)=\begin{array}{c}
\begin{vmatrix}
\underbrace{\begin{matrix}
a_0 & & & \\
a_1 & a_0 & & \\
a_2 & a_1 & a_0 & \\
& \ldots & & \\
& & & a_m
\end{matrix}}_{n \text{ columns}} & \underbrace{\begin{matrix}
b_0 & & & \\
b_1 & b_0 & & \\
b_2 & b_1 & b_0 & \\
& \ldots & & \\
& & & b_m
\end{matrix}}_{m \text{ columns}}\\
&
\end{vmatrix}
\end{array}
\end{equation}
\end{document}
You can use nicematrix
. Some work is needed to pull the braces nearer the matrix, but not so much: the idea is to use a phantom matrix to set the width for the underbrace; then it is raised up.
\documentclass{article}
\usepackage{amsmath}
\usepackage{nicematrix}
\usepackage{lipsum} % mock text to show spacing around the display
\begin{document}
\lipsum[1]
\begin{equation}
R(f,g)=
\begin{vNiceMatrix}[last-row]
a_0 & & & & b_0 & & & \\
a_1 & a_0 & & & b_1 & b_0 & & \\
a_2 & a_1 & a_0 & & b_2 & b_1 & b_0 & \\
\multicolumn{3}{c}{\cdots} & & \multicolumn{3}{c}{\cdots} & \\
& & & a_m & & & & b_n \\
\multicolumn{4}{c}{%
\raisebox{1ex}{$
\underbrace{%
\hphantom{\begin{matrix} a_0 & a_1 & a_2 & a_m\end{matrix}}
}_{\text{$n$ columns}}
$}%
}
&
\multicolumn{4}{c}{%
\raisebox{1ex}{$
\underbrace{%
\hphantom{\begin{matrix} b_0 & b_1 & b_2 & b_n\end{matrix}}
}_{\text{$m$ columns}}
$}%
}
\end{vNiceMatrix}
\end{equation}
\lipsum[2]
\end{document}
Note the \multicolumn
instructions in the body to avoid misalignments.
I would place the two underbraced explanatory expressions outside the vertical bars. This may be done by with the help of 2 "\smash
ed" matrix
environments and two judiciously chosen typographic struts -- one inside the \left\vert ... \right\vert
group, containing a matrix with 5 columns, and one outside that group, containing a matrix with 7 columns (and a \smash[t]
directive).
\documentclass{article}
\usepackage{amsmath}
\begin{document}
\begin{equation}
R(f,g)=
\left\vert
\smash{\underbrace{\begin{matrix}
a_0 & & & \\
a_1 & a_0 & & \\
a_2 & a_1 & a_0 & \\
& \cdots \\
& & & a_m
\end{matrix}}_{\text{$n$ columns}}}\;
\smash{\underbrace{\begin{matrix}
b_0 & & & \\
b_1 & b_0 & & \\
b_2 & b_1 & b_0 & \\
& \cdots \\
& & & b_n
\end{matrix}}_{\text{$m$ columns}}}
%% insert a first typographic strut:
\vphantom{\begin{matrix} 1\\2\\3\\4\\5 \end{matrix}}
\right\vert
%% insert a second typographic strut:
\smash[t]{\vphantom{\begin{matrix} 1\\2\\3\\4\\5\\6\\7 \end{matrix}}}
\end{equation}
\end{document}