trema (ddot) does not appear in mathrm mode with newtxmath font
With the default setup you need to use the text accents not math so \textrm{Möbius}
but \operatorname
gives "log-like" prefix operator spacing, so to get the effect of that, wrap in the primitive \mathop
command, so
\mathop{\textrm{Möbius}}
This works:
\documentclass[a4paper,10pt]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{newtxtext,newtxmath}
\DeclareMathOperator{\mobius}{M\mathnormal{\ddot{\mathrm{o}}}bius}
\begin{document}
$\mobius$
\end{document}