Lower overbrace in math environment

abraces can be used for this - swapping/mixing ofthe brace directions:

\documentclass{article}
\usepackage{abraces,mathtools}% http://ctan.org/pkg/{abraces,mathtools}
\begin{document}
\[
  \setbox9=\hbox{$(X \circ X)$}
  \operatorname{min} \big| 
    \mathrlap{\hspace{.5\wd9}\mathclap{\aunderbrace[L1U1R]{\scriptstyle\phantom{\text{Hadamard product}}}_{\text{Hadamard product}}}}
    (X \circ X) - (1 \ 2 \ 3 \ \cdots \ M )\big|^2
\]
\[
  \operatorname{min} \big| 
    \underbrace{(X \circ X)}_{\mathclap{\text{Hadamard product}}}
    - (1 \ 2 \ 3 \ \cdots \ M )\big|^2
\]
\end{document}

The use of \box9 is just for finding the correct width of (X \circ X). That is, some box movement is required in order to place the \aunderbrace at the correct location. The second option looks better though.

You could use an \overbrace as well. And, using \big (and friends) instead of \left...\right allows for a better appearance in terms of the absolute delimiters.

Here's a way that doesn't need box measurements. I define a command \setbelow with one optional and two mandatory arguments. With the optional argument, the stuff below can be lowered more if needed. The result is essentially the same as in Werner's answer:

For a more flexible solution that also works with \left and \right see this answer of mine.

\documentclass{article}
\usepackage{mathtools}
\newcommand*\setbelow[3][0ex]{%
  \mathop{#2\rule[-#1]{0pt}{0pt}}_{\mathclap{#3}}%
  }
\begin{document}
\[
  \min \bigl|
  \bigl( X \mathbin{\setbelow[0.2ex]{\circ}
                    {\overbrace{\scriptstyle\text{Hadamard product}}}} X \bigr)
  - \begin{pmatrix} 1 & 2 & 3 & \dots & M \end{pmatrix}
  \bigr|^2
\]
\end{document}

Note that you can simply write \min instead of \operatorname{min}!