Understanding \@ifnextchar
This is a very important function in the LaTeX kernel. The macro
\@ifnextchar
takes three argument. The first one should be a single token, usually [ but not necessarily.
When the input stream has the following tokens
\@ifnextchar<token>{<true>}{<false>}
TeX will look at the next token (skipping spaces) and compare it to the <token> which is the first argument to \@ifnextchar and will not remove it from the input stream.
If the two tokens coincide (have the same meaning, to be precise), TeX will use the <true> code, otherwise the <false> code.
The old fashioned way to define commands with an optional and a mandatory argument was something like
\newcommand{\xyz}{\@ifnextchar[{\@xyz}{\@xyz[default]}}
\def\@xyz[#1]#2{do something with #1 and #2}
where default is the default value for the optional argument. Nowadays we'd say
\newcommand\xyz[2][default]{something with #1 and #2}
(which eventually will do the same working as the old fashioned definition, but in a safer way). If the call is
\xyz[a]{b}
the test of \@ifnextchar would be true, so TeX would expand this into
\@xyz[a]{b}
because the [ is not removed as would be an argument. With
\xyz{b}
the test would be false, so the tokens would be replaced by
\@xyz[default]{b}
keeping TeX happy with respect to the definition of \@xyz.
The same applies in your case, with the difference that the <false> text has something more in it:
\@ifnextchar[\@myitem{\@noitemargtrue\@myitem[\@itemlabel]}}[aaa] bbbwill become\@myitem[aaa] bbb\@ifnextchar[\@myitem{\@noitemargtrue\@myitem[\@itemlabel]}} bbbwill become\@noitemargtrue\@myitem[\@itemlabel] bbb
so supplying a suitable argument between square brackets to \@myitem and setting a conditional.
Since the second argument to \@ifnextchar, in this case, is a single token, it doesn't need braces around it. The code
\@ifnextchar[{\@myitem}{\@noitemargtrue\@myitem[\@itemlabel]}}
would be completely equivalent.
See this answer for an explanation of how \@ifnextchar works internally in terms of \futurelet and this other one for a description of \futurelet (both by Martin Scharrer).
\@ifnextchar is a LaTeX conditional that peeks ahead at the following character. So, \@ifnextchar[ looks ahead to see if the following character in the input stream is a [ (opening left bracket). If this is true, then it executes the immediately following token, otherwise, it skips it and executes the token following that.
The first token, executed upon a true evaluation of \@ifnextchar[ is \@myitem. If the evaluation is false, it executes {\@noitemargtrue\@myitem[\@itemlabel]}} - the following group or "token", which also calls \@myitem now with an optional argument set [\@itemlabel]. Before calling \@myitem, the boolean \if@noitemarg is set to true (\@noitemargtrue).
The general idea behind this expression is to condition on whether an optional argument is supplied or not, and therefore avoid an error if you pass arguments to a macro where the parameter text does not match what is given. Using the above example, \@mymacro is defined using
\def\@myitem[#1]{<replacement text>}
Since this uses \def for the definition, the "optional" argument is actually not optional, but required. That's why the above usage is necessary. A more conventional way of conditioning is supplied using
\newcommand{\@myitem}[1][<arg>]{<replacement text>}
where <arg> specifies the value of the optional argument if it is not explicitly specified. xparse provides similar conditioning that, for the upper-level user, might be more intuitive.
You wish to know what specifically the line
\@ifnextchar[ \@myitem{\@noitemargtrue\@myitem[\@itemlabel]}}
does?
Let's introduce better linebreaking:
\@ifnextchar[% <-This is the 1st argument.
\@myitem% <-This is the 2nd argument.
{\@noitemargtrue\@myitem[\@itemlabel]}% <-This is the 3rd argument.
}%<- This is the next non-space token following the three arguments.
\@ifnextchar takes three non-optional arguments and then "looks" at the next non-space-token following these three arguments, leaving it (but not the space tokens if present) in place.
If the meaning of that non-space-token equals the meaning of the token delivered in the first argument, \@ifnextchar will deliver the second argument. Otherwise it will deliver the third argument.
With your code snippet, \@ifnextchar will find out that the opening bracket's meaning is not the same as the closing brace's meaning and will therefore deliver the 3rd argument, \@noitemargtrue\@myitem[\@itemlabel] while leaving the closing brace in place.
In the end you get:
\@noitemargtrue\@myitem[\@itemlabel]}
I don't know where that closing brace comes from.
Seems you overlooked the need of removing it when copying the snippet from Gonzalo Medina's answer where the following macro definition occurs:
\def\myitem{%
\@ifnextchar[ \@myitem{\@noitemargtrue\@myitem[\@itemlabel]}}
With that definition expanding \myitem yields:
\@ifnextchar[ \@myitem{\@noitemargtrue\@myitem[\@itemlabel]}
Which means that the meaning of the next non-space-token behind \myitem will be compared to the meaning of the opening-bracket-of-category-code-12-character-token.
In case the meanings are equal, \@myitem will be placed in front of that non-space-token, yielding \@myitem[... .
In case the meanings differ, \@noitemargtrue\@myitem[\@itemlabel] will be placed in front of that non-space-token, yielding \@noitemargtrue\@myitem[\@itemlabel]<whatsoever token whose meaning does not equal opening-bracket>...
Be that as it may—\@ifnextcharworks as follows:
\@ifnextchar<undelimited argument 1 containing a <single token>>%
<undelimited argument 2>%
<undelimited argument 3>%
<optional space tokens>
<non space token>
If meaning of <single token> from <undelimited argument 1> = meaning of <non space token>, then deliver <undelimited argument 2> and leave <non space token> in place, yielding:
<undelimited argument 2><non space token>.
If meaning of <single token> from <undelimited argument 1> =/= meaning of <non space token>, then deliver <undelimited argument 3> and leave <non space token> in place, yielding:
<undelimited argument 3><non space token>.
Be aware that \@ifnextchar performs many assigments. Therefore it cannot be used in pure-expansion-contexts (e.g. inside \csname..\endcsname) as expansion takes place in TeX's mouth while assignments take place in TeX's stomach.
Be aware that even outside pure-expansion-contexts \@ifnextchar cannot be used for reliably finding out whether the following <non space token> is the same token as the <single token> from <undelimited argument 1>. You can only find out whether those tokens have the same meaning.
Be aware that \@ifnextchar cannot be used for finding out whether a space token does follow. For one thing space tokens behind the third argument are optional and therefore will be discarded silently instead of becoming objects of comparison. For another thing the <single token> from the <undelimited argument 1 containing a <single token>> being a space token yields unpredictable behavior due to a flaw in the definition of \@ifnextchar.
E.g., try:
\documentclass{article}
\makeatletter
\begin{document}
\def\reserved@a#1\@ifnch{Let's bail out!}
\@ifnextchar{ }{some }{no }space?
\end{document}
Also be aware that <undelimited argument 1 containing a <single token>> is an undelimited argument. With undelimited arguments catcode-1-character-tokens and catcode-2-character-tokens (i.e. opening brace-character-tokens and closing brace-character-tokens) must be balanced. Therefore you cannot pass single brace tokens.
If people wish to check whether a brace token follows, they often do the workaround \let\bgroup={ or \let\egroup=} and
\@ifnextchar\bgroup{an opening brace follows}{something differing from opening brace follows} respective \@ifnextchar\egroup{a closing brace follows}{something differing from closing brace follows}.
But this workaround is not reliable because:
\@ifnextchar\bgroup{an opening brace follows}{something differing from opening brace follows}{ → an opening brace follows{
and
\@ifnextchar\bgroup{an opening brace follows}{something differing from opening brace follows}\bgroup → an opening brace follows\bgroup
respective
\@ifnextchar\egroup{a closing brace follows}{something differing from a closing brace follows}} → a closing brace follows}
and
\@ifnextchar\egroup{a closing brace follows}{something differing from a closing brace follows}\egroup → a closing brace follows\egroup
Often people use \@ifnextchar for testing with tokens that are not to be read from the input file but that are delivered as macro arguments.
Be aware that there is an easy way of reliably testing if the first token of an undelimited macro argument is a catcode-1-character-token (an opening brace)—the test can also be applied in expansion contexts:
\long\def\UDfirstoftwo#1#2{#1}%
\long\def\UDsecondoftwo#1#2{#2}%
%%----------------------------------------------------------------------
%% Check whether argument's first token is a catcode-1-character
%%......................................................................
%% \UDCheckWhetherBrace{<Argument which is to be checked>}%
%% {<Tokens to be delivered in case that argument
%% which is to be checked has leading
%% catcode-1-token>}%
%% {<Tokens to be delivered in case that argument
%% which is to be checked has no leading
%% catcode-1-token>}%
\long\def\UDCheckWhetherBrace#1{%
\romannumeral0\expandafter\UDsecondoftwo\expandafter{\expandafter{%
\string#1.}\expandafter\UDfirstoftwo\expandafter{\expandafter
\UDsecondoftwo\string}\expandafter\expandafter\UDfirstoftwo{ }{}%
\UDfirstoftwo}{\expandafter\expandafter\UDfirstoftwo{ }{}\UD@secondoftwo}%
}%
The gist of this test is attaching a dot to the argument for ensuring non-emptiness and then "hitting" with \string and forking whether \string hit an opening brace or something else.
In order to make things more obvious, I will change the linebreaking and add a bit of commenting:
\long\def\UDfirstoftwo#1#2{#1}%
\long\def\UDsecondoftwo#1#2{#2}%
%%----------------------------------------------------------------------
%% Check whether argument's first token is a catcode-1-character
%%......................................................................
%% \UDCheckWhetherBrace{<Argument which is to be checked>}%
%% {<Tokens to be delivered in case that argument
%% which is to be checked has leading
%% catcode-1-token>}%
%% {<Tokens to be delivered in case that argument
%% which is to be checked has no leading
%% catcode-1-token>}%
\long\def\UDCheckWhetherBrace#1{%
\romannumeral0% \romannumeral-expansion is used for ensuring
% that there will be no expansion-step in
% between whereafter unbalanced \if..\else..\fi remain.
\expandafter\UDsecondoftwo %<-This is the "important \UDSecondoftwo"
\expandafter{%
\expandafter{%
\string#1.% \string might hit an opening brace or something else.
% In any case \string cannot hit a closing brace as for one
% thing #1 in any case is brace balanced and for another thing
% the trailing dot catches up the case of #1 being empty.
}%<-This is the end of important \UDSecondoftwo's
% first argument in case #1's first token is an opening brace.
\expandafter\UDfirstoftwo %<-This is the "interesting \UDfirstoftwo".
\expandafter{%
\expandafter\UDsecondoftwo
\string}%
% The following will terminate \romannumeral's search for digits
% after having `\romannumeral` carrying out \UDfirstoftwo:
\expandafter\expandafter\UDfirstoftwo{ }{}\UDfirstoftwo
}%<-This is the end of important \UDSecondoftwo's
% first argument in case #1 is empty or #1's first token is
% not an opening brace.
% This also is the end of interesting \UDfirstoftwo's first
% argument in case #1's first token is an opening brace.
{%
% The following will terminate \romannumeral's search for digits
% after having `\romannumeral` carrying out \UDsecondoftwo:
\expandafter\expandafter\UDfirstoftwo{ }{}\UD@secondoftwo
}%
}%
With this test
\UDCheckWhetherBrace{text}{leading brace}{no leading brace} yields: no leading brace
\UDCheckWhetherBrace{{text}}{leading brace}{no leading brace} yields: leading brace
\UDCheckWhetherBrace{{}text}{leading brace}}{no leading brace} yields: leading brace