Extra long braces with \begin{array}[t]

Here are two alternatives to your current setup:

1. Put the construction inside another array.

2. Use cases instead and a different layout.

\documentclass{article}

\usepackage{amsmath,mathrsfs}

\begin{document}

\[
  \mathscr F = \left\{
    \begin{array}{@{} l @{}}
      \{B_{\alpha}\}_{\alpha\in J}:
        \begin{array}[t]{r @{~} l}
          \textit{i})   &  B_{\alpha} \text{ are open}\\
          \textit{ii})  &  \{B_{\alpha}\} \text{ is finitely inadequate}\\
          \textit{iii}) &  I \subset J \text{ and } A_{\alpha}=B_{\alpha} \text{ whenever } \alpha\in I
        \end{array}
    \end{array}
  \right\}
\]

\[
  \mathscr F =
    \{B_{\alpha}\}_{\alpha\in J}:
    \begin{cases}
      \textit{i})   &  B_{\alpha} \text{ are open}\\
      \textit{ii})  &  \{B_{\alpha}\} \text{ is finitely inadequate}\\
      \textit{iii}) &  I \subset J \text{ and } A_{\alpha}=B_{\alpha} \text{ whenever } \alpha\in I
    \end{cases}
\]

\end{document}

I propose two other variants, built upon Bmatrix , \parbox and an inline enumerate environment:

\documentclass{article}

\usepackage{amsmath,mathrsfs}
\usepackage[inline]{enumitem}

\begin{document}

\[
  \mathscr F =
    \begin{Bmatrix}
      \{B_{\alpha}\}_{\alpha\in J}:\:
        \parbox{75mm}{%
        \begin{enumerate*}[label =$ \roman*) $, itemjoin ={; \enspace}]
          \item $ B_{\alpha} $ are open
          \item $ \{B_{\alpha}\} $ is finitely inadequate
          \item$ I \subset J $ and $ A_{\alpha}=B_{\alpha} $ whenever $ \alpha\in I $.
        \end{enumerate*}}
    \end{Bmatrix}
 \]

\[
  \mathscr F =
    \begin{Bmatrix}
      \{B_{\alpha}\}_{\alpha\in J}:\;
        \parbox[t]{75mm}{%
        \begin{enumerate*}[label =$ \roman*) $, itemjoin ={; \enspace}]
          \item $ B_{\alpha} $ are open
          \item $ \{B_{\alpha}\} $ is finitely inadequate
          \item$ I \subset J $ and $ A_{\alpha}=B_{\alpha} $ whenever $\alpha\in I$.
        \end{enumerate*}}
    \end{Bmatrix}
 \]

\end{document}