How to make an equation more compact?

I recommend you do the following:

• Get rid of all \left and \right directives since they (a) insert undesirable horizontal whitespace and (b) don't actually do anything useful in the present context.

• Use \substack (provided by the amsmath package) to break the line below \bigcup into two parts.

• Use \textup (or \textnormal) instead of the default math italics in the definitions of the variable names "ref" and "new".

• Use \mathclap (provided by the mathtools package) to suppress whitespace to the left and right of \bigcup. (This step is also featured in @karlkoeller's answer.)

\documentclass{article}
\usepackage{mathtools,amssymb}
\newcommand\vnew{\textup{new}}
\newcommand\vref{\textup{ref}}
\begin{document}
Before: 
\begin{equation}
RA_U  \left ( new , ref \right ) =
\bigcup_{
  \left\{
    i  \text{ such that }  f_i \left(  ref \right) < f_i \left( new \right)
  \right\}
}
\left\{
  X \in \mathbb{R}^M ;
  x_i \in \left [ f_i \left(  ref \right) ; f_i \left( new \right) \right ]
\right\}
\end{equation}

After:
\begin{equation}
RA_U (\vnew,\vref) =
\bigcup_{ \mathclap{ \substack{i\text{ such that}\\f_i(\vref) < f_i(\vnew)}} } \,
\bigl\{ X\in\mathbb{R}^M ; x_i \in [ f_i(\vref) ; f_i(\vnew) ] \bigr\}
\end{equation}
\end{document}

You can use \mathclap from mathtools package:

\documentclass{article}

\usepackage[cmex10]{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}

\begin{document}
\begin{equation}
RA_U  ( \mathit{new}, \mathit{ref} ) =
\bigcup_{\mathclap{
  \left\{
    i  \text{ such that }  f_i (\mathit{ref}) < f_i ( \mathit{new} )
  \right\}
}}
\left\{
  X \in \mathbb{R}^M ;
  x_i \in [ f_i ( \mathit{ref} ) ; f_i ( \mathit{new} ) ]
\right\}
\end{equation}
\end{document} 

Output:

This is not new with respect to other answers, but I'd like to introduce a couple of tricks:

\documentclass{article}

\usepackage{amsmath,mathtools}
\usepackage{amssymb}

\newcommand{\V}[1]{\mathrm{#1}} % or \mathit
\newcommand{\numberset}[1]{\mathbb{#1}}
\newcommand{\R}{\numberset{R}}

\begin{document}
This is the original equation
\begin{equation}
RA_{U}(\V{new},\V{ref}) =
\bigcup_{
  \quad
    \mathclap{\substack{
    \text{$i$ such that}\\[\jot]
    f_i(\V{ref}) < f_i(\V{new})
  }}
  \quad
}
\{ 
  X \in \R^{M} : 
  x_i \in [f_i(\V{ref}); f_i(\V{new})] 
\}
\end{equation}

\renewcommand{\V}[1]{\mathit{#1}}
\renewcommand{\numberset}[1]{\mathsf{#1}}

Just to see the effect of changing just a couple of commands
\begin{equation}
RA_{U}(\V{new},\V{ref}) =
\bigcup_{
  \quad
    \mathclap{\substack{
    \text{$i$ such that}\\[\jot]
    f_i(\V{ref}) < f_i(\V{new})
  }}
  \quad
}
\{ 
  X \in \R^{M} : 
  x_i \in [f_i(\V{ref}); f_i(\V{new})] 
\}
\end{equation}
\end{document}

The main trick is the macro \V for typesetting multiletter variables; then also \numberset is defined, which is used indirectly for defining \R (and other number sets). I show how, just changing those definitions, one can change the output.

For the wide subscript I use \substack in a \mathclap, but adding some space around it in order that the union symbol is not adjacent to its surroundings.

Note the \jot used for giving some room between the two lines in the subscript. Also \text{$i$ such that} is handier to type than i\text{ such that}, because spaces are ‘natural’.