Why can't I multiply size by integer in calc?

The ordering of the arguments of the \setlength instruction matters. Your code has

\newlength{\foo}
\newcommand{\zzz}{5}
\setlength{\foo}{\zzz * 2in}

TeX starts with a scalar number (5) and is told to multiply it by a length (2in). Such on-the-fly type changes -- here: from scalar to length variable -- aren't allowed.

In contrast, the syntax of the expression

\setlength{\foo}{2in * \zzz}

is perfectly acceptable, as TeX starts with a length (2in) and multiplies it by a scalar (5), resulting in another length (10in).

\documentclass{article}
\usepackage{calc}
\newlength{\foo}
\newcommand{\zzz}{5}
\setlength{\foo}{ 2in * \zzz }
\usepackage[paperwidth=\foo,paperheight=\foo]{geometry}
\begin{document}
\end{document}