Latex custom counters

You was getting an error because of an extra space inside \csname...\endcsname...

When you do \csname cs\endcsname you define a control sequence \cs because TeX ignores spaces after control sequence names. But when you do \csname cs \endcsname, the space after cs is considered and the control sequence you create is actually \cs␣ (␣ denotes a space).

---

Fun (?) fact: A lot of commands use a space after the name of the control sequence to store a "secret" version of the command. For example, if you create a document and put \show\cite in it, you'll see:

> \cite=macro:
->\protect \cite  .

then you'll ask: but then I cannot see the definition of \cite? Yes, you can! Notice that there is a space after the \cite in the second line. To see the \cite␣ macro you do:

\expandafter\show\csname cite \endcsname

with a space after cite :)

---

Also, there was an extra space after α, β, γ, and δ in \c@greekalpha. I removed them for you. This space was not causing an error but it would appear when printed in the pdf.

\documentclass[11pt]{article}
\usepackage{amsfonts}
\usepackage{mathspec}
\setmathsfont(Digits,Latin,Greek)[Uppercase=Regular,Lowercase=Regular,]        {Bookman Old Style}
\setmainfont{Bookman Old Style}
\usepackage{enumitem}

\begin{document}
\newcounter{greekctr}\stepcounter{greekctr}\stepcounter{greekctr}

% \tracingonline=1
% \tracingmacros=1

\makeatletter%                                    V no space here!
    \def\greekc#1{\expandafter\@greekc\csname c@#1\endcsname}
    \def\@greekc#1{\ifcase #1 \or α\or β\or γ\or δ\else\@ctreerr \fi}
\makeatother

\makeatletter
    \newcommand{\greekalpha}[1]{\c@greekalpha{#1}}
    \newcommand{\c@greekalpha}[1]{%
        {%                                v    v    v    v And no spaces here
            \ifcase\number\value{#1} \or α\or β\or γ\or δ\or\fi
        }%
    }
\makeatother
\noindent

\begin{itemize}
    \item Value of counter: \thegreekctr
    \item Alph counter :\alph{greekctr}
    \item greekalpha: \greekalpha{greekctr}
    \item greekc: \greekc{greekctr}
\end{itemize}

Working use in enumerate environment:
\AddEnumerateCounter*{\greekalpha}{\c@greekalpha}{γ}
\begin{enumerate}[label=\greekalpha*]
    \item First
    \item Second
    \item Third
    \item Fourth
\end{enumerate}
\end{document}

The problem is with \csname c@#1 \endcsname. There is a space after #1, so for example if #1 is section this results in \c@section with a final space in the name of the command sequence. This is not defined, so TeX finds an undefined control sequence instead of a number which leads to the error. To fix this you have to delete the space before \endcsname. The space after \csname is different. It is not included in the name of the command sequence because spaces after a command sequence like \csname are gobbled by TeX after functioning as a terminator of the name of the command sequence. It is even necessary, because otherwise \csnamec@ would be interpreted as one (undefined) command sequence.