Cannot use ampersand as argument in user defined command with optional arguments

You've found a dark corner in the alignment process. Let's see what happens.

1. When starting a cell in an alignment, TeX expands tokens in order to see whether \omit appears (it's used, for example, in \multicolumn for spanning columns).

2. The expansion of \optal[&=] is

   \@protected@testopt \optal \\optal {=}[&=]
   

   (note that \\optal is a single control sequence)

3. The expansion of \@protected@testopt is

   \ifx \protect \@typeset@protect \expandafter \@testopt \else \@x@protect #1\fi
   

   so #1 is \optal (which is removed from the input stream)

4. The conditional is true, so what remains in the input stream is

   \@testopt \\optal {=}[&=]
   

5. The expansion of \@testopt is

   #1#2->\kernel@ifnextchar [{#1}{#1[{#2}]}
   

   so #1 is \\optal and #2 is = (the braces are stripped off); what remains in the input stream is

   \kernel@ifnextchar[{\\optal}{\\optal[{=}]}[&=]
   

Up to now no unexpandable token has been found, so TeX is still expanding tokens; the expansion of \kernel@ifnextchar starts with \let and now TeX determines that no \omit was present. Then TeX proceeds to build the alignment cell. However, it grabs only everything *up to the next & at the same level, so the contents of the cell is

\kernel@ifnextchar[{\\optal}{\\optal[{=}]}[

and no closing ] for the optional argument is found in the cell. The error message says

Argument of \\optal has an extra }

because in the search for the closing ] TeX find a closing brace (it would be too long to explain where it comes from).

With Sigur's workaround, what remains is

\kernel@ifnextchar[{\\optal}{\\optal[{=}]}[{&=}]

and now the & is not at the same level of grouping, so the cell is not terminated; TeX finds the [ so it does

\\optal[{&=}]

and, since the expansion of \\optal is [#1]->a #1 b the braces around the argument are stripped off and

a &= b

is found, which terminates the cell at the right time.

Wow!

Protect it with braces

\optal[{&=}]

Note that you can add more symbols to the side of an equation.

\documentclass{report}
\usepackage{amssymb}
\usepackage{amsmath}
\newcommand{\optal}[1][=]{a #1 b}

\begin{document}
$\optal$ 
$\optal[ \neq ]$ 
\begin{align}
\optal
\end{align}
\begin{align}
\optal[{&=}]         & \optal[{&=}] + \int \\
\optal[{&=}] + \iint & \optal[{&=}] - \partial
\end{align}
\end{document}

You can hide the & from the argument scanner without having to put a {} group in every instance by using the {\ifnum0='} idiom used in all table macros. Note however that (unlike the explicit use of {} within the argument) this does introduce a {} group into the cell, which does not affect your example but could in principle (hence my modified example with +. Hiding teh & automatically without introducing a {} mathord atom is possible but harder.

\documentclass[a4paper, 12pt]{book}
\usepackage{amssymb}
\usepackage{amsmath}

\begin{document}



% new command
\newcommand{\optal}{{\ifnum0=`}\fi\xoptcal}
\newcommand\xoptcal[1][=]{\ifnum0=`{\fi} +a #1 b }
%trying it out
$\optal$ \\
$\optal[ \neq ]$ \\
% trying it out inside align
\begin{align}
\optal[=]
\end{align}
%so far, so good
% trying it out with & as parameter
\begin{align}
\optal[&=]\\
+a&=b3\\
{}+a&=b3\\
\end{align}
% Fails with "Argument of \\optal has an extra } \end{align}"
% Paragraph ended before \\optal was complete \end{align}
\end{document}