Math subscript doesn't work in custom command

you're using the wrong syntax.

the optional argument should be in square brackets, not braces, and be input before the main argument, not after.

here's what you want: \LOC[1]{C}.

For optional arguments in curly braces and in a trailing position you need xparse. Also, you probably want to use \textsubscript.

\documentclass{article}

\usepackage{xparse}

\NewDocumentCommand\var{m}{\textsf{#1}}
\NewDocumentCommand\LOC{mG{}}{\var{LOC#1}\textsubscript{#2}}

\begin{document}

\LOC{C}---Without optional argument.

\var{LOCC}\textsubscript{1}---This looks correct.

\LOC{C}{1}---What happened here?

\end{document}

Your use of optional arguments doesn't match what is expected. Specifically, optional arguments use [..] while mandatory arguments use {..}. Also, \newcommand{<cmd>}[<num>][<opt>]{<stuff>} requires the optional argument to be placed before the mandatory one:

<cmd>[<opt>]{<mandatory>}...

If you wish to have the input you're currently using - specifying the optional argument at the end - you can use the following options:

\documentclass{article}

\makeatletter
\newcommand{\var}[1]{\textsf{#1}}
\newcommand{\LOCa@}[1][\relax]{\ifx#1\relax\else\ensuremath{_{#1}}\fi}
\newcommand{\LOCa}[1]{\var{LOC#1}\LOCa@}
\makeatother

\usepackage{xparse}

\NewDocumentCommand{\LOCb}{m o}{%
  \var{LOC#1}%
  \IfValueT{#2}{\ensuremath{_{#2}}}%
}

\begin{document}

\LOCa{C}---Without optional argument.

$\var{LOCC}_{1}$---This looks correct.

\LOCa{C}[1]---What happened here?

\LOCb{C}---Without optional argument.

$\var{LOCC}_{1}$---This looks correct.

\LOCb{C}[1]---What happened here?

\end{document}

The xparse interface is easier to understand. Additionally, consider coding your macros without the forced switches (to math mode) in mind. Rather switch to math mode where you use it within your document:

$\LOC{C}[1]$

Reference:

• When not to use \ensuremath for math macro?