How to define a command with two optional arguments?

You should start from examining #2 rather than #1, because #2 only can have a value if #1 has, too.

\documentclass{article}
\usepackage{xparse}

\NewDocumentCommand \C { o o m } {%
  \IfNoValueTF{#2}
   {%
    \IfNoValueTF {#1}
     {%
      C_{#3}%
     }
     {%
      C_{#3}^{[#1]}%
     }%
   }
   {%
    C_{#3, #1}^{[#2]}%
   }% 
}

\begin{document}

$\C{a} \quad \C[b]{a} \quad \C[c][b]{a}$ 

\end{document}

And don't use \DeclareDocumentCommand to override the meaning of \c: you'll regret it when you have to cite some French or Turkish author with “ç” in their name.