How to make the this brace bigger? "}"

I would suggested aligned (or gathered) for the two blocks on either side of the brace. In any case you can use a rule of zero width to give a minimum size to the brace. In the following the brace size has been made unnaturally large for demonstration purposes.

\documentclass{article}

\usepackage{amsmath}

\begin{document}

\begin{equation*}
  \left .
    \begin{aligned}
      2 \beta_{r,j}^{(l)} &= \beta_{r,j}^{l-1} + \beta_{R+r,j}^{(l-1)}
      \epsilon_{m}^{j}
      \\[1ex]
      2 \beta_{r,m+j}^{(l)} &= \beta_{r,j}^{l-1} +
      \beta_{R+r,j}^{(l-1)} \epsilon_{m}^{j}
    \end{aligned}
    \quad \vrule width 0pt height 7ex \right \}\quad
  \begin{aligned}
    &r=0,\dots,R-1 \\
    &j=0,\dots,m-1 \\
    &\epsilon_m = e^{\frac{-2 \pi i}{2^m}}
  \end{aligned}
\end{equation*}

\end{document}

You can use array for LHS also. And the vertical distance can be changed by \\[<dimen>].

\documentclass{article}
\usepackage{amsmath}
\begin{document}
\begin{equation*}
    \left .
    \begin{array}{r}
    2 \beta_{r,j}^{(l)} = \beta_{r,j}^{l-1} + \beta_{R+r,j}^{(l-1)} \epsilon_{m}^{j}
        \\[10pt]
    2 \beta_{r,m+j}^{(l)} = \beta_{r,j}^{l-1} + \beta_{R+r,j}^{(l-1)} \epsilon_{m}^{j}
    \end{array}
    \right \}
    \begin{array}{l}
        r=0,\dots,R-1
        \\
        j=0,\dots,m-1
        \\
        \epsilon_m = e^{\frac{-2 \pi i}{2^m}}
    \end{array}
\end{equation*}
\end{document}