Annotating a binary relation
\documentclass{article}
\usepackage{amsmath}
\begin{document}
$a \hat{\mathrel{\mathcal R}} b$
$a \mathrel{\hat{\mathcal R}} b$
$a \mathrel{\hat{\mathrel{\mathcal R}}} b$ Use this one
\end{document}
The hat is placed over the calligraphic R in an “optimal” way, according to the font designer's wishes.
If you want to have a different placement, more central, just add an empty subformula, so the hat is placed over the whole thing and not the single letter.
\documentclass{article}
\usepackage{amsmath}
\begin{document}
$a\mathrel{\hat{\mathcal{R}}}b$
$a\mathrel{\hat{\mathcal{R}{}}}b$
\end{document}
When TeX has to place a math accent, there are two cases.
First case: a single letter/symbol. The accent is placed over the “visual center” of the letter/symbol taking into account the slant of the font from which the letter/symbol is taken. This is how \hat{A}
moves the hat right so it sits above the vertex of A.
Second case: a subformula, that is, math mode material enclosed in braces, or a \left
-\right
pair, or the argument to the accent comamand itself if it doesn't turn out to be a single math symbol; also something like \mathrel{<symbol>}
is not considered to be a single math symbol. The accent is placed in the middle of the subformula.
With \hat{\mathcal{R}{}}
we’re in the second case, because the argument to \hat
isn’t a single letter/symbol, but also contains an empty subformula.
Thus \mathrel{\hat{\mathrel{\mathcal{R}}}}
works, but is not really the best solution.