Errors when trying to define a custom align environment

I'm not sure why but you can't use \begin{align} and \end{align} in the definition of a new environment; you have to use the "lower-level" macros \align and \endalign instead. Edit: as pointed by alexwlchan in his comment, you can find more details about that in section 6 of Technical notes on the amsmath package.

Here I've used the equivalent of an align* environment (see Herbert's answer to Define a custom align, and align* environment).

Note that you will get an error if you try to reset your grammarcounter at each chapter in the article class, because the latter doesn't have chapters; \section is the most high-level sectioning command in the article class. Did you mean

\numberwithin{grammarcounter}{section}

instead?

\documentclass{article}

\usepackage{amsmath}

\newcounter{grammarcounter}[section]

\makeatletter
\newenvironment{grammar}
{%
  \refstepcounter{grammarcounter}
  \start@align\@ne\st@rredtrue\m@ne
  \tag{$\Gamma_{\thegrammarcounter}$}
}{%
  \endalign
}
\makeatother
 
\begin{document}

\section{Foo}

\begin{grammar}
\label{gr:label}
E &::= E + E \\
E &::= a     \\
E &::= b     
\end{grammar}

\begin{grammar}
\label{gr:label2}
E &::= E + E \\
E &::= a     \\
E &::= b     
\end{grammar}

\end{document}

You can nest aligned or split in the grammar environment:

\documentclass[a4paper,11pt]{report}
\usepackage[fleqn,tbtags]{amsmath}

\newcounter{grammarcounter}[chapter]
\renewcommand{\thegrammarcounter}{\thechapter.\arabic{grammarcounter}}

\newenvironment{grammar}
  {\refstepcounter{grammarcounter}
   \begin{equation*}
   \tag{$\Gamma_{\thegrammarcounter}$}}
  {\end{equation*}}

\begin{document}

\chapter{Start}

\begin{grammar}
\label{gr:label}
\begin{aligned}
E &::= E + E \\
E &::= a
\end{aligned}
\end{grammar}

\begin{grammar}
\label{gr:label-two}
\begin{split}
E &::= E + E \\
E &::= a
\end{split}
\end{grammar}

\end{document}

Using split without the tbtags option is equivalent to aligned (the number will be vertically centered). So I presented two examples to show the difference, adding the option.

I wouldn't make grammar into an automatically multiline environment, as spacing considerations are involved. You can define a mlgrammar environment, if you want:

\newenvironment{mlgrammar}
  {\begin{grammar}\begin{aligned}}
  {\end{aligned}\end{grammar}}