define set of sets

Here are my suggestions:

• Use \Set and \SET commands such that you cannot forget braces and the formatting is consistent. Both take two arguments, where \Set typesets the second argument in math mode and \SET in text mode.

• Split the definition into two lines. It will be hard to read once you have inserted the proper conditions. It is particularly confusing to define T_f within the definition of B.

• Explain the meaning of B and T_f also verbally.

Here is an example assigning phantasy meaning.

\documentclass{article}
\usepackage{mathtools}
\newcommand\Set[2]{\{\,#1\mid#2\,\}}
\newcommand\SET[2]{\Set{#1}{\text{#2}}}
\begin{document}
\noindent
The boundary, $B$, is the collection of trust sets for all green functions:
\begin{align*}
B  &\coloneqq \Set{T_f}{\operatorname{color}(f)=\operatorname{green}}\\
\intertext{where}
T_f&\coloneqq \SET{t}{$t$ satisfies some condition depending on $f$}
\end{align*}
is the set of trustworthy points of function~$f$.
\end{document}

Same idea as @gernot’s, but following the guidelines given in Section 3.6 of the manual of the mathtools package:

\documentclass[a4paper]{article}
\usepackage[T1]{fontenc}
\usepackage{mathtools}
\usepackage{amssymb} % only because one of the examples uses "\mathbb"

% The portion between "\makeatletter" and "\makeatoyher" defines the abstract 
% command "\Set", together with some other ancillary commands (cf. the manual 
% of the `mathtools' package, p. 27).
\makeatletter

\newcommand*\SetSuchThat{\@ifstar\@SetSuchThat@star\@SetSuchThat@nostar}
\newcommand*\@SetSuchThat@star{%
    \mathrel{}%
    % \nobreak % superfluous inside "\left... ... \right..."
    \middle\vert
    \mathrel{}%
}
\newcommand*\@SetSuchThat@nostar[1][]{%
    \mathrel{#1\vert}%
}
\newcommand*\@SetSuchThat{}
\DeclarePairedDelimiterX \Set [2] {\lbrace}{\rbrace}
    {\nonscript\,#1\@SetSuchThat #2\nonscript\,}
\reDeclarePairedDelimiterInnerWrapper \Set {star} {%
    \mathopen{}%
    \mathclose\bgroup
    \def\@SetSuchThat{\SetSuchThat*}%
    #1#2\aftergroup\egroup #3
}
\reDeclarePairedDelimiterInnerWrapper \Set {nostar} {%
    \begingroup
        \def\@SetSuchThat{\SetSuchThat[\delimsize]}%
        \mathopen{#1}#2\mathclose{#3}%
    \endgroup
}

\makeatother



\begin{document}

Let us define:
\begin{equation}
    B = \Set % three arguments, the first is optional
            [\Big] % size specification for the delimiters
            {% typical set element; we nest onther "\Set" here
                T_{\!f} = \Set
                    % no optional argument: use ordinary-size delimiters
                    {t} % typical set element...
                    {\text{condition of $t$ in $T_{\!f}$}} % ...and its description
            }
            {\text{condition of $T_{\!f}$ in $B$}} % description for outer set
\end{equation}
In a in-line formula, though, I'd use smaller delimiters (\verb|\big| instead of
\verb|\Big|) for the outer set:
\(
    B = \Set % three arguments, the first is optional
            [\big] % size specification for the delimiters
            {% typical set element; we nest onther "\Set" here
                T_{\!f} = \Set
                    % no optional argument: use ordinary-size delimiters
                    {t} % typical set element...
                    {\text{condition of $t$ in $T_{\!f}$}} % ...and its description
            }
            {\text{condition of $T_{\!f}$ in $B$}} % description for outer set
\).
As you can see, however, the output is awful in any case!~(;-)

You can also replace the optional argument of the \verb|\Set| command with a
star~(\texttt{*}) for auto-sizing delimiters (that is,
\verb|\left\{|\,\ldots\verb+\middle|+\,\ldots\verb|\right\}|).  Example:
\begin{equation}
    A = \Set*
        {\frac{n}{n+1}}
        {n\in\mathbb{N}}
\end{equation}

\end{document}

After seeing the output

you should better appreciate the value of @gernot’s second suggestion, that is, split the definition into two separate pieces! :-)

Note: Personally, I insist on preferring a notation like

\Set{x}{\text{condition on $x$}}

over

\Set{x \given \text{condition on $x$}}

with appropriate definition of the \given command, as the manual of the mathtools package suggests (in other words, I prefer having two separate arguments for the typical element of the set and for the defining condition). Others don’t agree.

---

Update (December 12, 2019)

As noted in a comment, the mathtools package has been modified since this answer was posted, so that the code shown above doesn’t work any longer. With recent releases of that package, the following code should be used instead:

\documentclass[a4paper]{article}
\usepackage[T1]{fontenc}
\usepackage{mathtools}
\usepackage{amssymb} % only because one of the examples uses "\mathbb"

% The portion between "\makeatletter" and "\makeatoyher" defines the abstract 
% command "\Set", together with some other ancillary commands (cf. the manual 
% of the `mathtools' package, p. 27).
\makeatletter

\newcommand*\SetSuchThat{\@ifstar\@SetSuchThat@star\@SetSuchThat@nostar}
\newcommand*\@SetSuchThat@star{%
    \mathrel{}%
    % \nobreak % superfluous inside "\left... ... \right..."
    \middle\vert
    \mathrel{}%
}
\newcommand*\@SetSuchThat@nostar[1][]{%
    \mathrel{#1\vert}%
}
\newcommand*\@SetSuchThat{}
\DeclarePairedDelimiterX \Set [2] {\lbrace}{\rbrace}
    {\nonscript\,#1\@SetSuchThat #2\nonscript\,}
\reDeclarePairedDelimiterInnerWrapper \Set {star} {%
    \mathopen{}%
    \mathclose\bgroup
    \def\@SetSuchThat{\SetSuchThat*}%
    #1#2\aftergroup\egroup #3
}
\reDeclarePairedDelimiterInnerWrapper \Set {nostarnonscaled} {%
    \begingroup
        \def\@SetSuchThat{\SetSuchThat[\delimsize]}%
        \mathopen#1#2\mathclose#3%
    \endgroup
}
\reDeclarePairedDelimiterInnerWrapper \Set {nostarscaled} {%
    \begingroup
        \def\@SetSuchThat{\SetSuchThat[\delimsize]}%
        \mathopen{#1}#2\mathclose{#3}%
    \endgroup
}

\makeatother



\begin{document}

Let us define:
\begin{equation}
    B = \Set % three arguments, the first is optional
            [\Big] % size specification for the delimiters
            {% typical set element; we nest onther "\Set" here
                T_{\!f} = \Set
                    % no optional argument: use ordinary-size delimiters
                    {t} % typical set element...
                    {\text{condition of $t$ in $T_{\!f}$}} % ...and its description
            }
            {\text{condition of $T_{\!f}$ in $B$}} % description for outer set
\end{equation}
In a in-line formula, though, I'd use smaller delimiters (\verb|\big| instead of
\verb|\Big|) for the outer set:
\(
    B = \Set % three arguments, the first is optional
            [\big] % size specification for the delimiters
            {% typical set element; we nest onther "\Set" here
                T_{\!f} = \Set
                    % no optional argument: use ordinary-size delimiters
                    {t} % typical set element...
                    {\text{condition of $t$ in $T_{\!f}$}} % ...and its description
            }
            {\text{condition of $T_{\!f}$ in $B$}} % description for outer set
\).
As you can see, however, the output is awful in any case!~(;-)

You can also replace the optional argument of the \verb|\Set| command with a
star~(\texttt{*}) for auto-sizing delimiters (that is,
\verb|\left\{|\,\ldots\verb+\middle|+\,\ldots\verb|\right\}|).  Example:
\begin{equation}
    A = \Set*
        {\frac{n}{n+1}}
        {n\in\mathbb{N}}
\end{equation}

\end{document}

The output is exactly the same: