What is the difference of \mathop, \operatorname and \DeclareMathOperator?
When is it better to use \operatorname (or its wrapper \DeclareMathOperator) instead of \mathop? The answer is easy: always, unless you know precisely what's the behavior of \mathop.
First of all, one must of course recall that \operatorname and \DeclareMathOperator are provided by the amsopn package, which is automatically loaded by amsmath, but is also available standalone.
Let's see the main definitions:
\DeclareRobustCommand{\operatorname}{%
\@ifstar{\qopname\newmcodes@ m}%
{\qopname\newmcodes@ o}}%
\DeclareRobustCommand{\qopname}[3]{%
\mathop{#1\kern\z@\operator@font#3}%
\csname n#2limits@\endcsname}
\newcommand{\DeclareMathOperator}{%
\@ifstar{\@declmathop m}{\@declmathop o}}
\long\def\@declmathop#1#2#3{%
\@ifdefinable{#2}{%
\DeclareRobustCommand{#2}{\qopname\newmcodes@#1{#3}}}}
The command \qopname is not really meant to be used in a document; the purpose of its first argument is to add some declarations such as \newmcodes@ before typesetting the operator name.
The \operatorname and \DeclareMathOperator each have a *-variant that passes to \qopname a second argument m instead of o. The latter command is just a wrapper:
\DeclareMathOperator{\xyz}{xyz}
\DeclareMathOperator*{\XYZ}{XYZ}
are pretty much equivalent to saying
\DeclareRobustCommand{\xyz}{\operatorname{xyz}}
\DeclareRobustCommand{\XYZ}{\operatorname*{XYZ}}
(but treated more efficiently) so what we need to examine is just \operatorname.
A call of \operatorname{xyz} translates into
\qopname\newmcodes@ o {xyz}
which becomes
\mathop{\newmcodes@\kern\z@\operator@font xyz}\csname nolimits@\endcsname}
The
\newmcodes@declaration takes care that some characters are treated differently as usual in math formulas (in particular the hyphen doesn't become a minus sign);\kern\z@inserts an invisible object so that the entire contents of\mathopwill never be a single character (see https://tex.stackexchange.com/a/41267/4427 for details);\operator@fontschooses the predefined font for math operators, which usually is the upright text font;\nolimits@is just an alias for\nolimits, so following subscripts and superscripts won't be typeset above and below the operator, unless countermanded by a\limitsdeclaration.
For \operatorname* it's exactly the same, with the only difference that \nmlimits@ will be executed instead of \nolimits@. This macro is defined to be equivalent to \displaylimits, which makes the operator behave like the standard \lim or \min.
If an operator is used just a couple of times in a document, one can maybe dispense with defining a command with \DeclareMathOperator; but readability of the source is, in my opinion, enhanced if proper names for logical structures are used.
For \mathop one has to remember that it doesn't choose any particular font and it doesn't correct some glitches: so the output of
\mathop{\mathrm{abc-def}}
will be rather different from
\operatorname{abc-def}
and the latter is usually expected. In some rare cases \mathop is useful by itself:
\newcommand{\diff}{\mathop{}\!d}
is an example, where the empty \mathop is used to provide the correct spacing before the "d". Note that if one really wants to have an upright "d", it would be incorrect to define it as \operatorname{d} and the same as before with \mathrm{d} instead of d should be used: explicitly, the alternative is
\newcommand{\diff}{\mathop{}\!\mathrm{d}}
Also \stackrel and the better amsmath commands \overset and \underset are defined with an internal \mathop{...}\limits but wrapped up in some other construction.
\mathop is the most primitive command, declaring the class of a character or subformula to be a unary math operator like \sum, for instance. This has consequences for spacing, compared to binary operators declared with \mathbin or binary relations declared with \mathrel, for instance.
Most math operators are written out as textual names. Some definitions from latex.ltx:
\def\max{\mathop{\operator@font max}}
\def\min{\mathop{\operator@font min}}
\def\sup{\mathop{\operator@font sup}}
\def\inf{\mathop{\operator@font inf}}
\def\arg{\mathop{\operator@font arg}\nolimits}
\def\ker{\mathop{\operator@font ker}\nolimits}
\def\dim{\mathop{\operator@font dim}\nolimits}
\def\hom{\mathop{\operator@font hom}\nolimits}
You see that a special font \operator@font is used; furthermore, one can see that an operator declared with \mathop is usually typeset with limits, that is, super- and subscripts are placed in a special position in displaymode. This can be turned off with \nolimits.
\operatorname from the AMSTeX bundle encapsulates this as a user command:
\DeclareRobustCommand{\operatorname}{%
\@ifstar{\qopname\newmcodes@ m}%
{\qopname\newmcodes@ o}}%
\DeclareRobustCommand{\qopname}[3]{%
\mathop{#1\kern\z@\operator@font#3}%
\csname n#2limits@\endcsname}
This basically means that \operatorname{dual} will typeset a math operator "dual" which looks like \arg (i.e. without limits) and \operatorname*{dual} will typeset a math operator "dual" which looks like \max (i.e. with limits).
\DeclareMathOperator is the corresponding declaration, i.e. \DeclareMathOperator\dual{dual} creates a new macro \dual which works like \max.
On usage:
- With
\mathop, anything can be declared a math operator, i.e. if you want to use*like a prefix operator (with limits), you can just say\mathop{*}. \DeclareMathOperatorshould be used if you want to define a new operator like\max.\operatornameshould be used for special cases where you need a special operator name only once.
\mathop is a plainTeX command, which declares a big operator like
\newcommand{\Res}{\mathop{\hbox{Res}}}
DeclareMathOperator is from AmsMath's amsopn subpackage, called as:
\DeclareMathOperator{\Res}{Res}
whereas operatorname is kind of an operator "on-the-fly". Input of
\operatorname{Res}
has the same meaning as \Res defined with mathop or DeclareMathOperator
I think, if you loaded AmSMath (always a good idea) then I would use DeclareMathOperator and operatorame in cases this operator is seldom used in the file or to make some text in math-mode upright and with space-padding.