pdflatex and \hyphenation

Set \righthyphenmin to 2, because amsart uses the value 3. Then lemma will be hyphenated. And words including a dash are not hyphenated. Insert a \hskip0pt. See example.

\documentclass[12pt]{amsart}
\newcommand{\calG}{\mathcal{G}}
\newcommand{\calP}{\mathcal{P}}
\righthyphenmin=2
\begin{document}
\subsection{Projection to holes is coarsely Lipschitz}
The following lemma is used repeatedly throughout the paper.
\begin{itemize}
\item
for any hole $X$ for $\calG$, the projection $\pi_X(\calP)$ is an
$A$--\hskip0pt{}unparameterized quasi-geodesic and
\item
foobar.
\end{itemize}
\end{document}

TeX won't hyphenate words without leaving at least \lefthyphenmin letters before the hyphen and \righthyphenmin letters after it. These two parameters are set on a per language basis; for English, the typographic traditions require

\lefthyphenmin=2
\righthyphenmin=3

so it's immaterial if you say

\hyphenation{lem-ma}

in the document's preamble: the hyphenation point will not be considered anyway. It's interesting to know that, setting \righthyphenmin=2, the command

\showhyphens{lemma}

shows lem-ma on the terminal.

While globally setting \righthyphenmin=2 will solve the particular problem, I wouldn't recommend it, since it may add many improper hyphenation points in the rest of the document.

A "local" solution, that is, inputting lem\-ma at that spot is, in my opinion, the way to go: an explicit discretionary hyphen overrides the "minimum hyphenation rules" for that word.

Words containing an explicit or discretionary hyphen (- or \-) are possibly split only at the explicit hyphens: you can solve the problem with $A$-unparameterized by inserting \hspace{0pt}:

$A$-\hspace{0pt}unparameterized

or, maybe,

$A$\mbox{-}\hspace{0pt}unparameterized

so that TeX won't break after the explicit hyphen. Leave all these adjustments for the final stage of production, when you're sure that the text and the page parameters are in definitive form.