How can I change the "References" to "Reference" in the thebibliography environment?

It depends on the documentclass.

\renewcommand\bibname{Reference}

works for some.

As chl points out. Others use \refname instead of \bibname.

In my case, I added the following option to \printbibliography:

\printbibliography[title={Reference}]

For more details, see the BibLaTeX documentation, section 3.7.2 The Bibliography.

It seems \refname works fine for me while \bibname not.

For TOC, in case you need it, use \contentsname

\documentclass[10pt,a4paper]{article}
\renewcommand{\contentsname}{whatever}
\renewcommand{\refname}{whatever}