How to decrease spacing before chapter title?

For many reasons, titlesec continues to use the default \@makechapterhead macro for typesetting the chapter title when the chapter style is display. So

\documentclass[12pt]{report}
\usepackage{titlesec}
\titleformat{\chapter}[display]   
{\normalfont\huge\bfseries}{\chaptertitlename\ \thechapter}{20pt}{\Huge}   
\titlespacing*{\chapter}{0pt}{-50pt}{40pt}
\begin{document}
\chapter{Characters}

The origin of the group determinant begins with Richard Dedekind in the late 1800's. He began to explore the idea
after studying the discriminant in a normal field [A]. He made several observations about the group determinant, but
was only able to prove some of them.

\end{document}

will do, since \@makechapterhead adds a 50pt space above the title and 40pt after it.

A different strategy might be to redefine \@makechapterhead yourself:

\documentclass[12pt]{report}

\makeatletter
\def\@makechapterhead#1{%
  %%%%\vspace*{50\p@}% %%% removed!
  {\parindent \z@ \raggedright \normalfont
    \ifnum \c@secnumdepth >\m@ne
        \huge\bfseries \@chapapp\space \thechapter
        \par\nobreak
        \vskip 20\p@
    \fi
    \interlinepenalty\@M
    \Huge \bfseries #1\par\nobreak
    \vskip 40\p@
  }}
\def\@makeschapterhead#1{%
  %%%%%\vspace*{50\p@}% %%% removed!
  {\parindent \z@ \raggedright
    \normalfont
    \interlinepenalty\@M
    \Huge \bfseries  #1\par\nobreak
    \vskip 40\p@
  }}
\makeatother

\begin{document}
\chapter{Characters}

The origin of the group determinant begins with Richard Dedekind in the late 1800's. He began to explore the idea
after studying the discriminant in a normal field [A]. He made several observations about the group determinant, but
was only able to prove some of them.

\end{document}

little update for the memoir (which is based on book, so it should work in both):

the simple line \setlength{\beforechapskip}{20pt} works like a charm with no need for extra packages. \beforechapskip is used by \chapterheadstart which in turn is used by \makechapterhead as the \vspace{...} above the heading.