Mysterious horizontal space when using \label after beginning a user-defined environment

This is the case because \label was created in such a way that it doesn't introduce a second white space after it and doesn't change the spacefactor (a TeX internal variable which specifies how much the current space might be stretched to get alignment, it is set dependent on the last character prior to that space). This is done by \@bsphack and \@esphack. Unfortunately this ends the scope of \ignorespaces as there was a non-space token, and the space after \label{} is in fact inserted because there was no space prior to \label (else there would be an \ignorespaces inserted by \@esphack).

The following checks whether exercise is followed by a \label and if so eats the space after \label{}:

\documentclass{article}
\RequirePackage{expl3}
\RequirePackage{xparse}

\ExplSyntaxOn
\makeatletter
\newcounter{exercise}
\NewDocumentEnvironment{exercise}{}
{
  \refstepcounter{exercise}
  \par\noindent
  \theexercise.\hspace{4pt}X
  \@ifnextchar\label
    {\eatlabel}{\ignorespaces}
}
{
  \par
  \ignorespacesafterend
}
\newcommand\eatlabel{}
\def\eatlabel\label#1{\label{#1}\ignorespaces}
\makeatother
\ExplSyntaxOff

\begin{document}

\begin{exercise}
  If $H$ is a subgroup of $G$, then something.
\end{exercise}

\begin{exercise} \label{A}
  If $H$ is a subgroup of $G$, then something.
\end{exercise}

\end{document}

You have to delay the typesetting until possibly \label has appeared. Assuming these exercises aren't inside lists, the simplest way is using \everypar.

\documentclass{article}
\usepackage{expl3}
\usepackage{xparse}
\usepackage{showframe}

\ExplSyntaxOn
\newcounter{exercise}
\NewDocumentEnvironment{exercise}{}
 {
  \refstepcounter{exercise}
  \par
  \everypar={{\setbox0=\lastbox}\theexercise.\hspace{4pt}\everypar={}}
 }
 {
  \par
  \ignorespacesafterend
 }
\ExplSyntaxOff

\begin{document}

\begin{exercise}
  If $H$ is a subgroup of $G$, then something.
\end{exercise}

\begin{exercise}\label{A}
  If $H$ is a subgroup of $G$, then something.
\end{exercise}

\end{document}

With {\setbox0=\lastbox} we remove the indentation box; since \label doesn't start a paragraph, the space after it is ignored.

Possibly simpler: use a theorem-like environment.

\documentclass{article}
\usepackage{amsthm}
\usepackage{showframe}

\newtheoremstyle{exercise}
  {0pt}       % ABOVESPACE
  {0pt}       % BELOWSPACE
  {\upshape}  % BODYFONT
  {0pt}       % INDENT (empty value is the same as 0pt)
  {}          % HEADFONT
  {.}         % HEADPUNCT
  { }         % HEADSPACE
  % CUSTOM-HEAD-SPEC follows
  {\thmnumber{#2}}

\theoremstyle{exercise}
\newtheorem{exercise}{}

\begin{document}

\begin{exercise}
  If $H$ is a subgroup of $G$, then something.
\end{exercise}

\begin{exercise}\label{A}
  If $H$ is a subgroup of $G$, then something.
\end{exercise}

\end{document}