Is there a mathematical notation for restricting the depth of a factorial?

Yes, this is called a "falling factorial":

$$ n^{\underline{3}} = n(n-1)(n-2) $$ for example.

I pronounce $n^{\underline{3}}$ as "$n$ fall $3$".

Alternatively, you could write $$ \frac{5!}{2!} \, , $$ with the $2!$ cancelling out the unwanted terms. This is actually quite common. For example, it can be used to interpret the '$n$ choose $r$' formula: $$ {n \choose r} = \frac{n!}{r!(n-r)!} \, . $$ Say you have $n$ objects and you want to pick $r$ of them. Then, you have $n$ choices at first, then $(n-1), (n-2), \ldots ,$ and finally $(n-r+1)$. This gives us the $$ \frac{n!}{(n-r)!} $$ part of the formula. Then, the $r!$ removes duplicates.

This is also sometimes called "permutations of 3 out of 5 things" and denoted 5P3. You can find this on TI-8x calculators, at least, as nPr in the MATH->PRB menu (so you'd type 5 nPr 3).

I have also seen the falling factorial denoted with brackets as $[5]_3$. I'd say any notation for this will have to be clarified when you use it, though.