"Argument of \reserved@a has an extra }" error when using non-English language in \title

The \begin macro which starts an environment is "fragile", which means it cannot be used inside a moving argument. The \title command expands its argument, and therefore fragile commands can't be used inside it.

See the following question for an explanation of this.

• What is the difference between Fragile and Robust commands?

So the solution is to not use the {bengali} environment inside the title.

(You could actually "protect" the environment by using \protect\begin{bengali} .... \protect\end{bengali}) but there is a simpler solution:

Just use \textbengali{}:

\documentclass{memoir}

\usepackage{polyglossia}
\setmainlanguage{english}
\setotherlanguage[numerals=Devanagari]{bengali}
\newfontfamily\bengalifont[]{Bangla MN}
\title{\textbengali{বইয়ের টাইটেল}
  Book Title}

\begin{document}

\maketitle
\end{document}