What does "$\cdots$" mean in $\frac{n(n-1)(n-2)\cdots(n-r+2)}{(r-1)!}a^{n-r+1}b^{r-1}$?

It means: "There are too many terms to write, but follow the obvious pattern to fill them in".

In your example, you subtract $1$ from a factor to get the next factor. I might read that aloud as "$n$ times $n-1$ times $n-2$ all the way down to $n-r+2$".

As another example, $$ 3 + 6 + 9 + \cdots + 3n $$ would indicate the sum of all positive multiples of $3$ less than or equal to $3n$.

Writing it as "$\cdots\;$", would be better than "$\dots\;$" . It indicates a product:

$$ \frac{n(n-1)(n-2) \cdots (n-r+2)}{(r-1)!}a^{n-r+1}b^{r-1} =\frac{\prod_{k=0}^{r-2} (n-k)}{(r-1)!}a^{n-r+1}b^{r-1} $$