New command only for math mode: problem with \S
Werner already told you the reason where your command fails: trying to \let commands to LaTeX-robust commands (I know there is a more detailed explanation around here on tex.sx somewhere but I'm unable to find it…)
An up to date LaTeX provides \NewCommandCopy (as well as \RenewCommandCopy and \DeclareCommandCopy) which you can use instead of \let and which will also give you the expected result with LaTeX-robust commands.
Your MWE with only minor changes:
\documentclass{article}
\DeclareDocumentCommand{\newmathcommand}{mO{0}m}{%
\expandafter\NewCommandCopy\csname old\string#1\endcsname{#1}%
\expandafter\newcommand\csname new\string#1\endcsname[#2]{#3}
\DeclareRobustCommand#1{%
\ifmmode
\expandafter\let\expandafter\next\csname new\string#1\endcsname
\else
\expandafter\let\expandafter\next\csname old\string#1\endcsname
\fi
\next
}%
}
\newmathcommand{\S}{{\mathbf{S}}}
\newmathcommand{\o}{{\mathbf{o}}}
\newmathcommand{\c}{{\mathbf{c}}}
\begin{document}
Test S: $\S$, \S. \par
Test o: $\o$, \o. \par
Test c: $\c$, \c{c}.
\end{document}
I'm not convinced, though, that having two different definitions for a command inside and outside of math is a good idea…
It's quite interesting to see where the infinite loop is started. First we look at the standard definition of \S
% latex.ltx, line 3801:
\DeclareRobustCommand{\S}{\ifmmode\mathsection\else\textsection\fi}
This means, under the current implementation, that there are actually two definitions involved; in a complicated way that's not necessary to explain, the above code does essentially something like
\def\S{\protect\S•}
\def\S•{\ifmmode\mathsection\else\textsection\fi}
where the bullet denotes a space which is part of the name of the second macro (so it's easy to guess that some complicated set up is necessary, but that's not the point here).
Now you want to redefine \S like
\let\old/S=\S
\def\new/S{\mathbf{S}}
\DeclareRobustCommand\S{\ifmmode\new/S\else\old/S\fi}
where, in order to avoid reading ambiguities, / denotes the backslash in the command name obtained from \string\S via
\expandafter\let\csname old\string\S\endcsname=\S
OK, this is the same as doing
\def\old/S{\protect\S•}
\def\new/S{\mathbf{S}}
\def\S{\protect\S•}
\def\S•{\ifmmode\new/S\else\old/S\fi}
We're starting to see something fishy, aren't we? Let's see what happens if \S is found in math mode: each line is what results from macro expansion and command execution of the preceding line; here \protect is the same as \relax.
\S % start
\protect\S• % expansion
\S• % \protect is \relax and disappears
\ifmmode\new/S\else\old/S\fi % expansion
\new/S\else\old/S\fi % we're in math mode
\mathbf{S}\else\old/S\fi % expansion
\else\old/S\fi % \mathbf{S} is executed and disappears
\fi % \else gobbles everything up to \fi
The final expansion is empty. Good! We get \mathbf{S}!
Now let's see what happens in text mode:
\S % start
\protect\S• % expansion
\S• % \protect is \relax and disappears
\ifmmode\new/S\else\old/S\fi % expansion
\old/S\fi % we're not in math mode, tokens up to \else are gobbled
\protect\S•\fi % expansion
\S•\fi % \protect is \relax and disappears
\ifmmode\new/S\else\old/S\fi\fi % expansion
Sorry, infinite loop!
Your \let\old/S=\S only saves the “surface” definition of \S, not the “deep” one, which is the most important.
Why doesn't this happen with \c, for instance? Because it's defined (and made robust) in a completely different way
% latex.ltx, line 3732:
\DeclareTextAccentDefault{\c}{OT1}
and
% ot1enc.def, line 63:
\DeclareTextCommand{\c}{OT1}[1]
{\leavevmode\setbox\z@\hbox{#1}\ifdim\ht\z@=1ex\accent24 #1%
\else{\ooalign{\unhbox\z@\crcr\hidewidth\char24\hidewidth}}\fi}
With a newer LaTeX kernel, you can use \NewCommandCopy to overcome the problem. I also suggest a “simpler” reimplementation using expl3 features.
\documentclass{article}
%\usepackage{xparse} % not needed with LaTeX 2020-10-01 or later
\ExplSyntaxOn
\NewDocumentCommand{\newmathcommand}{mO{0}m}
{
\exp_args:Nc \NewCommandCopy {khue_kept_\cs_to_str:N #1} { #1 }
\exp_args:Nc \newcommand {khue_new_\cs_to_str:N #1}[#2]{#3}
\DeclareDocumentCommand {#1} {}
{
\mode_if_math:TF
{
\use:c {khue_new_\cs_to_str:N #1}
}
{
\use:c {khue_kept_\cs_to_str:N #1}
}
}
}
\ExplSyntaxOff
\newmathcommand{\S}{{\mathbf{S}}}
\newmathcommand{\c}{{\mathbf{c}}}
\newmathcommand{\o}{{\mathbf{o}}}
\begin{document}
Test a: $\c$, \c{c}
Test S: $\S$, \S.
Test o: $\o$, \o.
\end{document}
However, this is just for academic interest. Define \bS instead.
The problem is not in defining a command to do different things if called in text or math mode: as you see, the original definition of \S does exactly this! But it does so in order to get comparable output in the two cases. Your new \S does completely different things in text or math mode. Not a really good user interface, in my opinion.
The problem here stems from commands that are robust. Robust commands don't expand the way regular commands do, so your "old definition" ends up becoming cyclic and therefore runs in an endless loop.
One can test whether a command is defined as being robust using a technique described in How to use \CheckCommand with robust commands? and condition on storing the "expanded" robust command rather than its original definition.
\documentclass{article}
\usepackage{amsmath}
\usepackage{amssymb}
\makeatletter
\DeclareDocumentCommand{\newmathcommand}{ m O{0} m }{%
% Check whether command is robust or not (https://tex.stackexchange.com/a/63734/5764)
\ifcsname\expandafter\@gobble\string#1\space\endcsname
% Command is robust
\expandafter\expandafter\expandafter\let\expandafter\csname old\string#1\expandafter\endcsname\expandafter=\csname\expandafter\@gobble\string#1\space\endcsname
\else
% Command is not robust
\expandafter\let\csname old\string#1\endcsname=#1
\fi
\expandafter\newcommand\csname new\string#1\endcsname[#2]{#3}
\DeclareRobustCommand#1{%
\ifmmode
\expandafter\let\expandafter\next\csname new\string#1\endcsname
\else
\expandafter\let\expandafter\next\csname old\string#1\endcsname
\fi
\next
}%
}
\makeatother
\newmathcommand{\S}{{\mathbf{S}}}
\newmathcommand{\o}{{\mathbf{o}}}
\newmathcommand{\c}{{\mathbf{c}}}
\begin{document}
Test S: $\S$, \S \par
Test o: $\o$, \o \par
Test c: $\c$, \c{c}
\end{document}