newcommand vs. DeclareMathOperator

\DeclareMathOperator is designed to create commands that should typeset operator names such as sin and lim. Some of these are already defined in base TeX or LaTeX so one writes 2\sin\theta

instead of 2sin\theta

giving correct spacing and font. If you need an operator of this type that is not predefined, then you create it with \DeclareMathOperator, e.g. the space of endomorphisms of a vector space is written \End V

but you need to make the definition \DeclareMathOperator{\End}{End} first: a minimal working example is

\documentclass{article}
\usepackage{amsmath}
\DeclareMathOperator{\End}{End}
\begin{document}

\( \End V \)

\end{document}

\newcommand is much more general and can be used to define direct short cuts or more complicated macros. So for example if you find youself writing \left.\frac d{dt}\right|_{t=0} many times in your document you can package this up as a single command \dtzero via \newcommand{\dtzero}{\left.\frac d{dt}\right|_{t=0}} and just type \dtzero each time instead.

I would usually recommend reserving \DeclareMathOperator for the use described above and using \newcommand in most other situations. To get the effect of \DeclareMathOperator in a one-off situation, or inside a \newcommand, you use \operatorname; so you can write \operatorname{End}V for the above example.

Finally, one should note that there is a starred version \DeclareMathOperator*. This is used for defining operators that have limits typeset beneath them instead of to the right (at least when in a display). For example

Similarly there is the starred variant \operatorname*.

\documentclass{article}
\usepackage{amsmath}
\DeclareMathOperator{\End}{End}
\DeclareMathOperator*{\Max}{Max}
\begin{document}

\begin{displaymath}
  \Max_{x\in A} f(x) \qquad  \End_R V 
\end{displaymath}

\end{document}

Remark. The above code samples load the amsmath package. Strictly speaking all you need is the amsopn package, which amsmath reads in automatically. Alternatively, one can load mathtools which is a modern package building further on amsmath.

I usually use

\newcommand{\Ker}{\operatorname{Ker}}

instead of

\DeclareMathOperator{\Ker}{Ker}

However, the second is much practical.

Another difference is that \DeclareMathOperator can only be used in the preamble while \newcommand has no such restriction.