Why does the hyperref package change the paper size?

Just because the output format has a default paper size doesn't mean that LaTeX knows about it.

In the "old days", there was no communication between TeX and whatever printed/displayed the DVI file w.r.t. the size of the paper. Hence the reason for the defaults you mention — if you're American, you'll set up your page margins and so on for letter paper, so when you go to print/display the page it makes sense to show a letter-sized page.

Remember that LaTeX is intended to produce the same documents given the same input on different computers (packages changing their behaviour notwithstanding). So you must always declare the class option [a4paper] if you want your documents printed or displayed on A4-sized paper.

To answer your specific question, since pdfTeX there have been TeX commands to explicitly set and communicate the page size to the output driver; since LaTeX itself has been largely frozen it doesn't used these and the vanilla behaviour is to rely on the defaults you talk about to set the page sizes.

But hyperref is a modern package and refers to many PDF internals, and one of the things it also does is ensure that the paper size of the PDF matches the paper size that LaTeX used.

You can test the layout e.g. with the layout package.

\documentclass[a4paper]{article}
\usepackage{layout}
\begin{document}
\layout
\end{document}

This gives this values for letter (without a4paper):

and this with a4paper:

As you can see quite some values differ, most importantly paperwidth and paperheight.

Without hyperref, the pdf page width will be set with the default of your system. In your case this means the pdf paperwidth and the tex paperwidth are actually different.

hyperref (or graphicx) will force the pdf paperwidth in sync with your tex paperwidth.

It's wrong not to pass the a4paper in the first place.

Compare the two printouts of a simple \kant document. Can you see the difference? In the “no a4paper” case, the type block is placed in a wrong position and has the wrong dimensions. Specifically, the left margin is larger than the right margin, and the bottom margin is larger than what it should be.

\documentclass[
  a4paper
]{article}
\usepackage{kantlipsum}
\begin{document}
\kant[1-10]
\end{document}

Without a4paper

With a4paper